2024-02-05T04:40:36.403 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-05T04:40:36.404 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} --owner scheduled_nmordech@teuthology --verbose --name nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi --archive /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539668 -- /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539668/orig.config.yaml 2024-02-05T04:40:36.427 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-05T04:40:36.688 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539668 branch: wip-nitzan-max-pglog-size description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} email: null first_in_suite: false job_id: '7539668' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-nitzan-max-pglog-size ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 workunit: branch: wip-nitzan-max-pglog-size sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 owner: scheduled_nmordech@teuthology priority: 1000 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 8695 sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-nitzan-max-pglog-size suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 targets: smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEi9x4SB2ZkDPoRoPJVvBcM5XZ8KSrFdFl71VrJ7EllReZJGauGBsUzhTAR8jD1YjCMCg9xZfKb0PcnA6f3+8Tk= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-01-31_07:15:30 tube: smithi user: nmordech verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-05T04:40:36.689 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa; will attempt to use it 2024-02-05T04:40:36.689 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks 2024-02-05T04:40:36.690 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-05T04:40:36.691 INFO:teuthology.task.internal:Checking packages... 2024-02-05T04:40:36.715 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98' 2024-02-05T04:40:36.715 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-05T04:40:36.715 INFO:teuthology.packaging:ref: None 2024-02-05T04:40:36.715 INFO:teuthology.packaging:tag: None 2024-02-05T04:40:36.715 INFO:teuthology.packaging:branch: wip-nitzan-max-pglog-size 2024-02-05T04:40:36.715 INFO:teuthology.packaging:sha1: d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-05T04:40:36.716 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-nitzan-max-pglog-size 2024-02-05T04:40:36.858 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-658-gd4ed20cf-1jammy 2024-02-05T04:40:36.860 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-05T04:40:36.867 INFO:teuthology.task.internal:no buildpackages task found 2024-02-05T04:40:36.867 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-05T04:40:36.872 INFO:teuthology.task.internal:Saving configuration 2024-02-05T04:40:36.882 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-05T04:40:36.888 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T04:40:36.912 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539668', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 04:34:49.541630', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILJTHOLSRx3r/kHi/JBwoNijUy+FBOyFgh+dfHN2VGBl'} 2024-02-05T04:40:36.912 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-05T04:40:36.919 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-02-05T04:40:36.919 INFO:teuthology.run_tasks:Running task console_log... 2024-02-05T04:40:36.955 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f06b5c2d160>, signals=[15]) 2024-02-05T04:40:36.955 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-05T04:40:36.961 INFO:teuthology.task.internal:Opening connections... 2024-02-05T04:40:36.961 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2024-02-05T04:40:36.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:40:37.092 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-05T04:40:37.104 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2024-02-05T04:40:37.204 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2024-02-05T04:40:37.205 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2024-02-05T04:40:37.254 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-02-05T04:40:37.254 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="Ubuntu" 2024-02-05T04:40:37.254 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="22.04" 2024-02-05T04:40:37.254 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-02-05T04:40:37.254 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_CODENAME=jammy 2024-02-05T04:40:37.254 INFO:teuthology.orchestra.run.smithi148.stdout:ID=ubuntu 2024-02-05T04:40:37.254 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE=debian 2024-02-05T04:40:37.254 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://www.ubuntu.com/" 2024-02-05T04:40:37.254 INFO:teuthology.orchestra.run.smithi148.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-02-05T04:40:37.254 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-02-05T04:40:37.254 INFO:teuthology.orchestra.run.smithi148.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-02-05T04:40:37.254 INFO:teuthology.orchestra.run.smithi148.stdout:UBUNTU_CODENAME=jammy 2024-02-05T04:40:37.255 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2024-02-05T04:40:37.283 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-05T04:40:37.292 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-05T04:40:37.297 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-05T04:40:37.297 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2024-02-05T04:40:37.302 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-05T04:40:37.307 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-05T04:40:37.308 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2024-02-05T04:40:37.363 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-05T04:40:37.401 INFO:teuthology.run_tasks:Running task kernel... 2024-02-05T04:40:37.415 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-05T04:40:37.415 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-05T04:40:37.415 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-05T04:40:37.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:40:37.420 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-02-05T04:40:37.466 INFO:teuthology.orchestra.run.smithi148.stdout:5.15.0-56-generic 2024-02-05T04:40:37.466 INFO:teuthology.task.kernel:Running kernel on smithi148: 5.15.0-56-generic 2024-02-05T04:40:37.466 DEBUG:teuthology.orchestra.run.smithi148:> sudo apt-get clean 2024-02-05T04:40:37.591 DEBUG:teuthology.orchestra.run.smithi148:> sudo apt-get update 2024-02-05T04:40:37.757 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-05T04:40:37.796 INFO:teuthology.orchestra.run.smithi148.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-02-05T04:40:37.835 INFO:teuthology.orchestra.run.smithi148.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-02-05T04:40:37.940 INFO:teuthology.orchestra.run.smithi148.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-02-05T04:40:38.040 INFO:teuthology.orchestra.run.smithi148.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [560 kB] 2024-02-05T04:40:38.221 INFO:teuthology.orchestra.run.smithi148.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,325 kB] 2024-02-05T04:40:38.263 INFO:teuthology.orchestra.run.smithi148.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,109 kB] 2024-02-05T04:40:38.439 INFO:teuthology.orchestra.run.smithi148.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [268 kB] 2024-02-05T04:40:38.478 INFO:teuthology.orchestra.run.smithi148.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-02-05T04:40:38.480 INFO:teuthology.orchestra.run.smithi148.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,342 kB] 2024-02-05T04:40:38.609 INFO:teuthology.orchestra.run.smithi148.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [34.8 kB] 2024-02-05T04:40:38.610 INFO:teuthology.orchestra.run.smithi148.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [220 kB] 2024-02-05T04:40:38.624 INFO:teuthology.orchestra.run.smithi148.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-02-05T04:40:38.625 INFO:teuthology.orchestra.run.smithi148.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,042 kB] 2024-02-05T04:40:38.660 INFO:teuthology.orchestra.run.smithi148.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [394 kB] 2024-02-05T04:40:38.667 INFO:teuthology.orchestra.run.smithi148.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [207 kB] 2024-02-05T04:40:38.677 INFO:teuthology.orchestra.run.smithi148.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-02-05T04:40:38.678 INFO:teuthology.orchestra.run.smithi148.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.2 kB] 2024-02-05T04:40:38.679 INFO:teuthology.orchestra.run.smithi148.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,313 kB] 2024-02-05T04:40:38.768 INFO:teuthology.orchestra.run.smithi148.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [214 kB] 2024-02-05T04:40:38.778 INFO:teuthology.orchestra.run.smithi148.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-02-05T04:40:38.778 INFO:teuthology.orchestra.run.smithi148.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [838 kB] 2024-02-05T04:40:38.798 INFO:teuthology.orchestra.run.smithi148.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [686 kB] 2024-02-05T04:40:38.813 INFO:teuthology.orchestra.run.smithi148.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [589 kB] 2024-02-05T04:40:38.870 INFO:teuthology.orchestra.run.smithi148.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [235 kB] 2024-02-05T04:40:38.874 INFO:teuthology.orchestra.run.smithi148.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-02-05T04:40:38.875 INFO:teuthology.orchestra.run.smithi148.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-02-05T04:40:38.876 INFO:teuthology.orchestra.run.smithi148.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,164 B] 2024-02-05T04:40:38.876 INFO:teuthology.orchestra.run.smithi148.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-02-05T04:40:38.877 INFO:teuthology.orchestra.run.smithi148.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-02-05T04:40:38.878 INFO:teuthology.orchestra.run.smithi148.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2024-02-05T04:40:38.879 INFO:teuthology.orchestra.run.smithi148.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2024-02-05T04:40:38.880 INFO:teuthology.orchestra.run.smithi148.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2024-02-05T04:40:38.931 INFO:teuthology.orchestra.run.smithi148.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-02-05T04:40:38.990 INFO:teuthology.orchestra.run.smithi148.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [160 kB] 2024-02-05T04:40:38.996 INFO:teuthology.orchestra.run.smithi148.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.2 kB] 2024-02-05T04:40:39.076 INFO:teuthology.orchestra.run.smithi148.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2024-02-05T04:40:39.076 INFO:teuthology.orchestra.run.smithi148.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-02-05T04:40:39.077 INFO:teuthology.orchestra.run.smithi148.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-02-05T04:40:39.393 INFO:teuthology.orchestra.run.smithi148.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-02-05T04:40:39.398 INFO:teuthology.orchestra.run.smithi148.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-02-05T04:40:39.399 INFO:teuthology.orchestra.run.smithi148.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-02-05T04:40:39.413 INFO:teuthology.orchestra.run.smithi148.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-02-05T04:40:39.415 INFO:teuthology.orchestra.run.smithi148.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-02-05T04:40:43.931 INFO:teuthology.orchestra.run.smithi148.stdout:Fetched 11.2 MB in 2s (5,761 kB/s) 2024-02-05T04:40:45.250 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2024-02-05T04:40:45.267 DEBUG:teuthology.orchestra.run.smithi148:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-05T04:40:45.368 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2024-02-05T04:40:45.538 INFO:teuthology.orchestra.run.smithi148.stdout:Building dependency tree... 2024-02-05T04:40:45.538 INFO:teuthology.orchestra.run.smithi148.stdout:Reading state information... 2024-02-05T04:40:45.684 INFO:teuthology.orchestra.run.smithi148.stdout:The following packages were automatically installed and are no longer required: 2024-02-05T04:40:45.685 INFO:teuthology.orchestra.run.smithi148.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-05T04:40:45.685 INFO:teuthology.orchestra.run.smithi148.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-05T04:40:45.685 INFO:teuthology.orchestra.run.smithi148.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-05T04:40:45.685 INFO:teuthology.orchestra.run.smithi148.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-02-05T04:40:45.685 INFO:teuthology.orchestra.run.smithi148.stdout: python2.7-minimal 2024-02-05T04:40:45.685 INFO:teuthology.orchestra.run.smithi148.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-05T04:40:45.686 INFO:teuthology.orchestra.run.smithi148.stdout:The following additional packages will be installed: 2024-02-05T04:40:45.686 INFO:teuthology.orchestra.run.smithi148.stdout: linux-generic linux-headers-5.15.0-92 linux-headers-5.15.0-92-generic 2024-02-05T04:40:45.686 INFO:teuthology.orchestra.run.smithi148.stdout: linux-headers-generic linux-image-5.15.0-92-generic 2024-02-05T04:40:45.687 INFO:teuthology.orchestra.run.smithi148.stdout: linux-modules-5.15.0-92-generic linux-modules-extra-5.15.0-92-generic 2024-02-05T04:40:45.688 INFO:teuthology.orchestra.run.smithi148.stdout:Suggested packages: 2024-02-05T04:40:45.688 INFO:teuthology.orchestra.run.smithi148.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-02-05T04:40:45.688 INFO:teuthology.orchestra.run.smithi148.stdout:Recommended packages: 2024-02-05T04:40:45.688 INFO:teuthology.orchestra.run.smithi148.stdout: thermald 2024-02-05T04:40:45.740 INFO:teuthology.orchestra.run.smithi148.stdout:The following NEW packages will be installed: 2024-02-05T04:40:45.740 INFO:teuthology.orchestra.run.smithi148.stdout: linux-headers-5.15.0-92 linux-headers-5.15.0-92-generic 2024-02-05T04:40:45.740 INFO:teuthology.orchestra.run.smithi148.stdout: linux-image-5.15.0-92-generic linux-modules-5.15.0-92-generic 2024-02-05T04:40:45.740 INFO:teuthology.orchestra.run.smithi148.stdout: linux-modules-extra-5.15.0-92-generic 2024-02-05T04:40:45.741 INFO:teuthology.orchestra.run.smithi148.stdout:The following packages will be upgraded: 2024-02-05T04:40:45.742 INFO:teuthology.orchestra.run.smithi148.stdout: linux-generic linux-headers-generic linux-image-generic 2024-02-05T04:40:45.869 INFO:teuthology.orchestra.run.smithi148.stdout:3 upgraded, 5 newly installed, 0 to remove and 282 not upgraded. 2024-02-05T04:40:45.870 INFO:teuthology.orchestra.run.smithi148.stdout:Need to get 113 MB of archives. 2024-02-05T04:40:45.870 INFO:teuthology.orchestra.run.smithi148.stdout:After this operation, 582 MB of additional disk space will be used. 2024-02-05T04:40:45.870 INFO:teuthology.orchestra.run.smithi148.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-92-generic amd64 5.15.0-92.102 [22.5 MB] 2024-02-05T04:40:46.833 INFO:teuthology.orchestra.run.smithi148.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-92-generic amd64 5.15.0-92.102 [11.5 MB] 2024-02-05T04:40:47.055 INFO:teuthology.orchestra.run.smithi148.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-92-generic amd64 5.15.0-92.102 [63.8 MB] 2024-02-05T04:40:48.473 INFO:teuthology.orchestra.run.smithi148.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.92.89 [1,700 B] 2024-02-05T04:40:48.473 INFO:teuthology.orchestra.run.smithi148.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.92.89 [2,498 B] 2024-02-05T04:40:48.473 INFO:teuthology.orchestra.run.smithi148.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-92 all 5.15.0-92.102 [12.3 MB] 2024-02-05T04:40:49.358 INFO:teuthology.orchestra.run.smithi148.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-92-generic amd64 5.15.0-92.102 [2,859 kB] 2024-02-05T04:40:49.602 INFO:teuthology.orchestra.run.smithi148.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.92.89 [2,346 B] 2024-02-05T04:40:49.979 INFO:teuthology.orchestra.run.smithi148.stdout:Fetched 113 MB in 4s (29.3 MB/s) 2024-02-05T04:40:50.144 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package linux-modules-5.15.0-92-generic. 2024-02-05T04:40:52.194 INFO:teuthology.orchestra.run.smithi148.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-02-05T04:40:52.198 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../0-linux-modules-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-05T04:40:52.316 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-modules-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-05T04:40:54.918 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package linux-image-5.15.0-92-generic. 2024-02-05T04:40:54.939 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../1-linux-image-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-05T04:40:55.027 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-image-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-05T04:40:55.514 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-92-generic. 2024-02-05T04:40:55.530 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-05T04:40:55.563 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-modules-extra-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-05T04:41:02.187 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../3-linux-generic_5.15.0.92.89_amd64.deb ... 2024-02-05T04:41:02.306 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-generic (5.15.0.92.89) over (5.15.0.56.54) ... 2024-02-05T04:41:02.630 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.92.89_amd64.deb ... 2024-02-05T04:41:02.784 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-image-generic (5.15.0.92.89) over (5.15.0.56.54) ... 2024-02-05T04:41:03.117 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package linux-headers-5.15.0-92. 2024-02-05T04:41:03.140 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../5-linux-headers-5.15.0-92_5.15.0-92.102_all.deb ... 2024-02-05T04:41:03.186 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-headers-5.15.0-92 (5.15.0-92.102) ... 2024-02-05T04:41:09.304 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package linux-headers-5.15.0-92-generic. 2024-02-05T04:41:09.341 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../6-linux-headers-5.15.0-92-generic_5.15.0-92.102_amd64.deb ... 2024-02-05T04:41:09.443 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-headers-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-05T04:41:12.203 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.92.89_amd64.deb ... 2024-02-05T04:41:12.312 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking linux-headers-generic (5.15.0.92.89) over (5.15.0.56.54) ... 2024-02-05T04:41:12.712 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-headers-5.15.0-92 (5.15.0-92.102) ... 2024-02-05T04:41:12.833 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-headers-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-05T04:41:12.968 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-headers-generic (5.15.0.92.89) ... 2024-02-05T04:41:13.096 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-image-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-05T04:41:14.581 INFO:teuthology.orchestra.run.smithi148.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-02-05T04:41:14.581 INFO:teuthology.orchestra.run.smithi148.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-02-05T04:41:14.581 INFO:teuthology.orchestra.run.smithi148.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-92-generic 2024-02-05T04:41:14.581 INFO:teuthology.orchestra.run.smithi148.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-92-generic 2024-02-05T04:41:14.731 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-modules-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-05T04:41:16.191 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-modules-extra-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-05T04:41:17.609 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-image-generic (5.15.0.92.89) ... 2024-02-05T04:41:17.735 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up linux-generic (5.15.0.92.89) ... 2024-02-05T04:41:17.861 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for linux-image-5.15.0-92-generic (5.15.0-92.102) ... 2024-02-05T04:41:17.927 INFO:teuthology.orchestra.run.smithi148.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-02-05T04:41:17.927 INFO:teuthology.orchestra.run.smithi148.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-92-generic 2024-02-05T04:41:32.043 INFO:teuthology.orchestra.run.smithi148.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-02-05T04:41:32.043 INFO:teuthology.orchestra.run.smithi148.stdout:Sourcing file `/etc/default/grub' 2024-02-05T04:41:32.064 INFO:teuthology.orchestra.run.smithi148.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-05T04:41:32.088 INFO:teuthology.orchestra.run.smithi148.stdout:Generating grub configuration file ... 2024-02-05T04:41:32.379 INFO:teuthology.orchestra.run.smithi148.stdout:Found linux image: /boot/vmlinuz-5.15.0-92-generic 2024-02-05T04:41:32.396 INFO:teuthology.orchestra.run.smithi148.stdout:Found initrd image: /boot/initrd.img-5.15.0-92-generic 2024-02-05T04:41:32.713 INFO:teuthology.orchestra.run.smithi148.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-05T04:41:32.718 INFO:teuthology.orchestra.run.smithi148.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-05T04:41:32.780 INFO:teuthology.orchestra.run.smithi148.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-05T04:41:32.785 INFO:teuthology.orchestra.run.smithi148.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-05T04:41:32.957 INFO:teuthology.orchestra.run.smithi148.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-05T04:41:32.957 INFO:teuthology.orchestra.run.smithi148.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-02-05T04:41:32.957 INFO:teuthology.orchestra.run.smithi148.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-05T04:41:32.996 INFO:teuthology.orchestra.run.smithi148.stdout:done 2024-02-05T04:41:33.513 DEBUG:teuthology.orchestra.run.smithi148:> dpkg -s linux-image-generic 2024-02-05T04:41:33.537 INFO:teuthology.orchestra.run.smithi148.stdout:Package: linux-image-generic 2024-02-05T04:41:33.537 INFO:teuthology.orchestra.run.smithi148.stdout:Status: install ok installed 2024-02-05T04:41:33.537 INFO:teuthology.orchestra.run.smithi148.stdout:Priority: optional 2024-02-05T04:41:33.538 INFO:teuthology.orchestra.run.smithi148.stdout:Section: kernel 2024-02-05T04:41:33.538 INFO:teuthology.orchestra.run.smithi148.stdout:Installed-Size: 21 2024-02-05T04:41:33.538 INFO:teuthology.orchestra.run.smithi148.stdout:Maintainer: Ubuntu Kernel Team 2024-02-05T04:41:33.538 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: amd64 2024-02-05T04:41:33.538 INFO:teuthology.orchestra.run.smithi148.stdout:Source: linux-meta 2024-02-05T04:41:33.538 INFO:teuthology.orchestra.run.smithi148.stdout:Version: 5.15.0.92.89 2024-02-05T04:41:33.538 INFO:teuthology.orchestra.run.smithi148.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-92), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2024-02-05T04:41:33.538 INFO:teuthology.orchestra.run.smithi148.stdout:Depends: linux-image-5.15.0-92-generic, linux-modules-extra-5.15.0-92-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-05T04:41:33.538 INFO:teuthology.orchestra.run.smithi148.stdout:Recommends: thermald 2024-02-05T04:41:33.538 INFO:teuthology.orchestra.run.smithi148.stdout:Description: Generic Linux kernel image 2024-02-05T04:41:33.538 INFO:teuthology.orchestra.run.smithi148.stdout: This package will always depend on the latest generic kernel image 2024-02-05T04:41:33.538 INFO:teuthology.orchestra.run.smithi148.stdout: available. 2024-02-05T04:41:33.538 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-92-generic 2024-02-05T04:41:33.539 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-05T04:41:33.539 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-05T04:41:33.539 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-05T04:41:33.539 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi148.front.sepia.ceph.com, path=None, version=distro) 2024-02-05T04:41:33.539 DEBUG:teuthology.orchestra.run.smithi148:> sudo apt-get clean 2024-02-05T04:41:33.706 DEBUG:teuthology.orchestra.run.smithi148:> sudo apt-get update 2024-02-05T04:41:33.948 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-05T04:41:34.020 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-02-05T04:41:34.021 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-02-05T04:41:34.101 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-02-05T04:41:35.280 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2024-02-05T04:41:35.297 DEBUG:teuthology.orchestra.run.smithi148:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-02-05T04:41:35.355 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2024-02-05T04:41:35.543 INFO:teuthology.orchestra.run.smithi148.stdout:Building dependency tree... 2024-02-05T04:41:35.555 INFO:teuthology.orchestra.run.smithi148.stdout:Reading state information... 2024-02-05T04:41:35.723 INFO:teuthology.orchestra.run.smithi148.stdout:linux-image-generic is already the newest version (5.15.0.92.89). 2024-02-05T04:41:35.723 INFO:teuthology.orchestra.run.smithi148.stdout:The following packages were automatically installed and are no longer required: 2024-02-05T04:41:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-05T04:41:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-05T04:41:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-05T04:41:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-05T04:41:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-05T04:41:35.724 INFO:teuthology.orchestra.run.smithi148.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-05T04:41:35.762 INFO:teuthology.orchestra.run.smithi148.stdout:0 upgraded, 0 newly installed, 0 to remove and 282 not upgraded. 2024-02-05T04:41:35.764 DEBUG:teuthology.orchestra.run.smithi148:> dpkg -s linux-image-generic 2024-02-05T04:41:35.780 INFO:teuthology.orchestra.run.smithi148.stdout:Package: linux-image-generic 2024-02-05T04:41:35.780 INFO:teuthology.orchestra.run.smithi148.stdout:Status: install ok installed 2024-02-05T04:41:35.780 INFO:teuthology.orchestra.run.smithi148.stdout:Priority: optional 2024-02-05T04:41:35.780 INFO:teuthology.orchestra.run.smithi148.stdout:Section: kernel 2024-02-05T04:41:35.780 INFO:teuthology.orchestra.run.smithi148.stdout:Installed-Size: 21 2024-02-05T04:41:35.780 INFO:teuthology.orchestra.run.smithi148.stdout:Maintainer: Ubuntu Kernel Team 2024-02-05T04:41:35.780 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: amd64 2024-02-05T04:41:35.780 INFO:teuthology.orchestra.run.smithi148.stdout:Source: linux-meta 2024-02-05T04:41:35.780 INFO:teuthology.orchestra.run.smithi148.stdout:Version: 5.15.0.92.89 2024-02-05T04:41:35.780 INFO:teuthology.orchestra.run.smithi148.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-92), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2024-02-05T04:41:35.780 INFO:teuthology.orchestra.run.smithi148.stdout:Depends: linux-image-5.15.0-92-generic, linux-modules-extra-5.15.0-92-generic, linux-firmware, intel-microcode, amd64-microcode 2024-02-05T04:41:35.780 INFO:teuthology.orchestra.run.smithi148.stdout:Recommends: thermald 2024-02-05T04:41:35.781 INFO:teuthology.orchestra.run.smithi148.stdout:Description: Generic Linux kernel image 2024-02-05T04:41:35.781 INFO:teuthology.orchestra.run.smithi148.stdout: This package will always depend on the latest generic kernel image 2024-02-05T04:41:35.781 INFO:teuthology.orchestra.run.smithi148.stdout: available. 2024-02-05T04:41:35.781 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2024-02-05T04:41:35.825 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.J86CvLRf4v 2024-02-05T04:41:35.825 DEBUG:teuthology.orchestra.run.smithi148:> sudo cp /boot/grub/grub.cfg /tmp/tmp.J86CvLRf4v 2024-02-05T04:41:35.880 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.J86CvLRf4v 2024-02-05T04:41:35.994 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.J86CvLRf4v is 10KB 2024-02-05T04:41:36.008 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.J86CvLRf4v 2024-02-05T04:41:36.014 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-02-05T04:41:36.068 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-05T04:41:36.068 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-02-05T04:41:36.069 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-02-05T04:41:36.137 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-92-generic 2024-02-05T04:41:36.137 DEBUG:teuthology.orchestra.run.smithi148:> sudo update-grub 2024-02-05T04:41:37.424 INFO:teuthology.orchestra.run.smithi148.stderr:Sourcing file `/etc/default/grub' 2024-02-05T04:41:37.425 INFO:teuthology.orchestra.run.smithi148.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-02-05T04:41:37.429 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2024-02-05T04:41:37.631 INFO:teuthology.orchestra.run.smithi148.stderr:Found linux image: /boot/vmlinuz-5.15.0-92-generic 2024-02-05T04:41:37.636 INFO:teuthology.orchestra.run.smithi148.stderr:Found initrd image: /boot/initrd.img-5.15.0-92-generic 2024-02-05T04:41:37.881 INFO:teuthology.orchestra.run.smithi148.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-02-05T04:41:37.886 INFO:teuthology.orchestra.run.smithi148.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-02-05T04:41:37.924 INFO:teuthology.orchestra.run.smithi148.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-02-05T04:41:37.929 INFO:teuthology.orchestra.run.smithi148.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-02-05T04:41:38.037 INFO:teuthology.orchestra.run.smithi148.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-02-05T04:41:38.038 INFO:teuthology.orchestra.run.smithi148.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-02-05T04:41:38.038 INFO:teuthology.orchestra.run.smithi148.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-02-05T04:41:38.047 INFO:teuthology.orchestra.run.smithi148.stderr:done 2024-02-05T04:41:38.066 DEBUG:teuthology.orchestra.run.smithi148:> sudo shutdown -r now 2024-02-05T04:42:08.097 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-05T04:42:08.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-05T04:42:08.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:42:26.492 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-02-05T04:42:35.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-05T04:42:35.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:42:38.556 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-02-05T04:42:50.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-05T04:42:50.579 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:43:08.988 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-02-05T04:43:23.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-05T04:43:24.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:43:24.002 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-02-05T04:43:42.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-05T04:43:42.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:43:42.352 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-02-05T04:43:42.946 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-05T04:43:42.947 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-92-generic"... 2024-02-05T04:43:42.947 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-02-05T04:43:42.992 INFO:teuthology.orchestra.run.smithi148.stdout:5.15.0-92-generic 2024-02-05T04:43:42.993 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-92-generic vs 5.15.0-92-generic 2024-02-05T04:43:42.993 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-05T04:43:42.993 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-02-05T04:43:43.995 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-05T04:43:43.995 DEBUG:teuthology.orchestra.run.smithi148:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-05T04:43:44.073 INFO:teuthology.orchestra.run.smithi148.stdout:ttyS1 2024-02-05T04:43:44.102 DEBUG:teuthology.parallel:result is None 2024-02-05T04:43:44.102 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-05T04:43:44.110 INFO:teuthology.task.internal:Creating test directory... 2024-02-05T04:43:44.110 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-05T04:43:44.115 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-05T04:43:44.147 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-05T04:43:44.154 INFO:teuthology.task.internal:Creating archive directory... 2024-02-05T04:43:44.154 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-05T04:43:44.176 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-05T04:43:44.182 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-05T04:43:44.183 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-05T04:43:44.232 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:43:44.240 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:43:44.241 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-05T04:43:44.248 INFO:teuthology.task.internal:Configuring sudo... 2024-02-05T04:43:44.249 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-05T04:43:44.295 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-05T04:43:44.304 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-05T04:43:44.304 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-05T04:43:44.344 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T04:43:44.392 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T04:43:44.440 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-05T04:43:44.440 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-05T04:43:44.522 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2024-02-05T04:43:44.599 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-05T04:43:44.607 INFO:teuthology.task.internal:Starting timer... 2024-02-05T04:43:44.607 INFO:teuthology.run_tasks:Running task pcp... 2024-02-05T04:43:44.616 INFO:teuthology.run_tasks:Running task selinux... 2024-02-05T04:43:44.625 DEBUG:teuthology.task.selinux:Excluding smithi148: OS 'ubuntu' does not support SELinux 2024-02-05T04:43:44.625 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-05T04:43:44.626 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-02-05T04:43:44.626 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-05T04:43:44.626 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-05T04:43:44.635 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-05T04:43:44.653 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-05T04:43:44.655 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi148.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-05T04:50:38.932 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi148.front.sepia.ceph.com')] 2024-02-05T04:50:38.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-05T04:50:38.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:50:39.020 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-02-05T04:50:39.100 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-02-05T04:50:39.101 INFO:teuthology.run_tasks:Running task clock... 2024-02-05T04:50:39.112 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-05T04:50:39.112 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-05T04:50:39.113 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T04:50:39.171 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-02-05T04:50:39.171 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: Command line: ntpd -gq 2024-02-05T04:50:39.172 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: ---------------------------------------------------- 2024-02-05T04:50:39.172 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: ntp-4 is maintained by Network Time Foundation, 2024-02-05T04:50:39.172 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-02-05T04:50:39.172 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: corporation. Support and training for ntp-4 are 2024-02-05T04:50:39.172 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: available at https://www.nwtime.org/support 2024-02-05T04:50:39.172 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: ---------------------------------------------------- 2024-02-05T04:50:39.173 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: proto: precision = 0.060 usec (-24) 2024-02-05T04:50:39.173 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: basedate set to 2022-02-04 2024-02-05T04:50:39.173 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: gps base set to 2022-02-06 (week 2196) 2024-02-05T04:50:39.173 INFO:teuthology.orchestra.run.smithi148.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-05T04:50:39.173 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-02-05T04:50:39.173 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: restrict ::: KOD does nothing without LIMITED. 2024-02-05T04:50:39.174 INFO:teuthology.orchestra.run.smithi148.stderr:restrict ::: KOD does nothing without LIMITED. 2024-02-05T04:50:39.175 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: Listen and drop on 0 v6wildcard [::]:123 2024-02-05T04:50:39.175 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-02-05T04:50:39.175 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: Listen normally on 2 lo 127.0.0.1:123 2024-02-05T04:50:39.175 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: Listen normally on 3 enp3s0f1 172.21.15.148:123 2024-02-05T04:50:39.175 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: Listen normally on 4 lo [::1]:123 2024-02-05T04:50:39.175 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7257%5]:123 2024-02-05T04:50:39.175 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:39 ntpd[17076]: Listening on routing socket on fd #22 for interface updates 2024-02-05T04:50:40.172 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:40 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:40.172 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:40 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:40.172 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:40 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:40.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:40 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:40.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:40 ntpd[17076]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-05T04:50:40.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:40 ntpd[17076]: can't open /var/log/ntpstats/peerstats.20240205: Permission denied 2024-02-05T04:50:42.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:42 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:42.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:42 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:42.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:42 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:42.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:42 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:42.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:42 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:42.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:42 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:42.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:42 ntpd[17076]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-05T04:50:42.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:42 ntpd[17076]: can't open /var/log/ntpstats/peerstats.20240205: Permission denied 2024-02-05T04:50:42.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:42 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:42.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:42 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:42.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:42 ntpd[17076]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-05T04:50:42.174 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:42 ntpd[17076]: can't open /var/log/ntpstats/peerstats.20240205: Permission denied 2024-02-05T04:50:43.172 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:43 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:43.172 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:43 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:43.172 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:43 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:43.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:43 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:43.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:43 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:43.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:43 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:43.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:43 ntpd[17076]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-05T04:50:43.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:43 ntpd[17076]: can't open /var/log/ntpstats/peerstats.20240205: Permission denied 2024-02-05T04:50:44.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:44 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:44.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:44 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:44.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:44 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:44.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:44 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:44.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:44 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:44.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:44 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:44.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:44 ntpd[17076]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-05T04:50:44.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:44 ntpd[17076]: can't open /var/log/ntpstats/peerstats.20240205: Permission denied 2024-02-05T04:50:44.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:44 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:44.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:44 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:44.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:44 ntpd[17076]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-05T04:50:44.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:44 ntpd[17076]: can't open /var/log/ntpstats/peerstats.20240205: Permission denied 2024-02-05T04:50:45.172 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:45 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:45.172 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:45 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:45.172 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:45 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:45.172 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:45 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:45.172 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:45 ntpd[17076]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-05T04:50:45.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:45 ntpd[17076]: can't open /var/log/ntpstats/peerstats.20240205: Permission denied 2024-02-05T04:50:46.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:46 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:46.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:46 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:46.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:46 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:46.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:46 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:46.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:46 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:46.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:46 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:46.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:46 ntpd[17076]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-05T04:50:46.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:46 ntpd[17076]: can't open /var/log/ntpstats/peerstats.20240205: Permission denied 2024-02-05T04:50:46.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:46 ntpd[17076]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-02-05T04:50:46.173 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:46 ntpd[17076]: can't open /var/log/ntpstats/rawstats.20240205: Permission denied 2024-02-05T04:50:46.174 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:46 ntpd[17076]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-02-05T04:50:46.174 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:46 ntpd[17076]: can't open /var/log/ntpstats/peerstats.20240205: Permission denied 2024-02-05T04:50:46.174 INFO:teuthology.orchestra.run.smithi148.stdout: 5 Feb 04:50:46 ntpd[17076]: ntpd: time slew -0.004963 s 2024-02-05T04:50:46.174 INFO:teuthology.orchestra.run.smithi148.stdout:ntpd: time slew -0.004963s 2024-02-05T04:50:46.174 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:46 ntpd[17076]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-02-05T04:50:46.174 INFO:teuthology.orchestra.run.smithi148.stderr: 5 Feb 04:50:46 ntpd[17076]: can't open /var/log/ntpstats/loopstats.20240205: Permission denied 2024-02-05T04:50:46.227 INFO:teuthology.orchestra.run.smithi148.stdout: remote refid st t when poll reach delay offset jitter 2024-02-05T04:50:46.227 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================== 2024-02-05T04:50:46.227 INFO:teuthology.orchestra.run.smithi148.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-05T04:50:46.227 INFO:teuthology.orchestra.run.smithi148.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-05T04:50:46.227 INFO:teuthology.orchestra.run.smithi148.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-05T04:50:46.227 INFO:teuthology.orchestra.run.smithi148.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-02-05T04:50:46.228 INFO:teuthology.run_tasks:Running task install... 2024-02-05T04:50:46.237 DEBUG:teuthology.task.install:project ceph 2024-02-05T04:50:46.237 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98'}} 2024-02-05T04:50:46.237 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98'} 2024-02-05T04:50:46.237 INFO:teuthology.task.install:Using flavor: default 2024-02-05T04:50:46.243 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-05T04:50:46.243 INFO:teuthology.task.install:extra packages: [] 2024-02-05T04:50:46.243 DEBUG:teuthology.orchestra.run.smithi148:> sudo apt-key list | grep Ceph 2024-02-05T04:50:46.329 INFO:teuthology.orchestra.run.smithi148.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-02-05T04:50:46.358 INFO:teuthology.orchestra.run.smithi148.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-02-05T04:50:46.358 INFO:teuthology.orchestra.run.smithi148.stdout:uid [ unknown] Ceph.com (release key) 2024-02-05T04:50:46.358 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-02-05T04:50:46.359 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-05T04:50:46.503 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default/ 2024-02-05T04:50:46.504 INFO:teuthology.task.install.deb:Package version is 19.0.0-658-gd4ed20cf-1jammy 2024-02-05T04:50:46.614 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-05T04:50:46.614 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-02-05T04:50:46.628 DEBUG:teuthology.orchestra.run.smithi148:> sudo apt-get update 2024-02-05T04:50:46.853 INFO:teuthology.orchestra.run.smithi148.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy InRelease 2024-02-05T04:50:46.874 INFO:teuthology.orchestra.run.smithi148.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-02-05T04:50:46.890 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-02-05T04:50:46.896 INFO:teuthology.orchestra.run.smithi148.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy Release.gpg 2024-02-05T04:50:46.917 INFO:teuthology.orchestra.run.smithi148.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,404 B] 2024-02-05T04:50:46.923 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-02-05T04:50:46.945 INFO:teuthology.orchestra.run.smithi148.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.7 kB] 2024-02-05T04:50:47.000 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-02-05T04:50:47.082 INFO:teuthology.orchestra.run.smithi148.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-02-05T04:50:47.249 INFO:teuthology.orchestra.run.smithi148.stdout:Fetched 28.8 kB in 0s (65.6 kB/s) 2024-02-05T04:50:48.106 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2024-02-05T04:50:48.121 DEBUG:teuthology.orchestra.run.smithi148:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-658-gd4ed20cf-1jammy cephadm=19.0.0-658-gd4ed20cf-1jammy ceph-mds=19.0.0-658-gd4ed20cf-1jammy ceph-mgr=19.0.0-658-gd4ed20cf-1jammy ceph-common=19.0.0-658-gd4ed20cf-1jammy ceph-fuse=19.0.0-658-gd4ed20cf-1jammy ceph-test=19.0.0-658-gd4ed20cf-1jammy ceph-volume=19.0.0-658-gd4ed20cf-1jammy radosgw=19.0.0-658-gd4ed20cf-1jammy python3-rados=19.0.0-658-gd4ed20cf-1jammy python3-rgw=19.0.0-658-gd4ed20cf-1jammy python3-cephfs=19.0.0-658-gd4ed20cf-1jammy python3-rbd=19.0.0-658-gd4ed20cf-1jammy libcephfs2=19.0.0-658-gd4ed20cf-1jammy libcephfs-dev=19.0.0-658-gd4ed20cf-1jammy librados2=19.0.0-658-gd4ed20cf-1jammy librbd1=19.0.0-658-gd4ed20cf-1jammy rbd-fuse=19.0.0-658-gd4ed20cf-1jammy 2024-02-05T04:50:48.183 INFO:teuthology.orchestra.run.smithi148.stdout:Reading package lists... 2024-02-05T04:50:48.373 INFO:teuthology.orchestra.run.smithi148.stdout:Building dependency tree... 2024-02-05T04:50:48.373 INFO:teuthology.orchestra.run.smithi148.stdout:Reading state information... 2024-02-05T04:50:48.547 INFO:teuthology.orchestra.run.smithi148.stdout:The following packages were automatically installed and are no longer required: 2024-02-05T04:50:48.547 INFO:teuthology.orchestra.run.smithi148.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-02-05T04:50:48.547 INFO:teuthology.orchestra.run.smithi148.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-02-05T04:50:48.547 INFO:teuthology.orchestra.run.smithi148.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-02-05T04:50:48.547 INFO:teuthology.orchestra.run.smithi148.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-02-05T04:50:48.548 INFO:teuthology.orchestra.run.smithi148.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-02-05T04:50:48.548 INFO:teuthology.orchestra.run.smithi148.stdout:Use 'sudo apt autoremove' to remove them. 2024-02-05T04:50:48.548 INFO:teuthology.orchestra.run.smithi148.stdout:The following additional packages will be installed: 2024-02-05T04:50:48.549 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-02-05T04:50:48.549 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-02-05T04:50:48.549 INFO:teuthology.orchestra.run.smithi148.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-02-05T04:50:48.549 INFO:teuthology.orchestra.run.smithi148.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-02-05T04:50:48.550 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-02-05T04:50:48.550 INFO:teuthology.orchestra.run.smithi148.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-02-05T04:50:48.551 INFO:teuthology.orchestra.run.smithi148.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-02-05T04:50:48.552 INFO:teuthology.orchestra.run.smithi148.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-02-05T04:50:48.552 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-02-05T04:50:48.552 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet zip 2024-02-05T04:50:48.552 INFO:teuthology.orchestra.run.smithi148.stdout:Suggested packages: 2024-02-05T04:50:48.552 INFO:teuthology.orchestra.run.smithi148.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-02-05T04:50:48.552 INFO:teuthology.orchestra.run.smithi148.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-02-05T04:50:48.552 INFO:teuthology.orchestra.run.smithi148.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-02-05T04:50:48.552 INFO:teuthology.orchestra.run.smithi148.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-02-05T04:50:48.552 INFO:teuthology.orchestra.run.smithi148.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-02-05T04:50:48.552 INFO:teuthology.orchestra.run.smithi148.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-02-05T04:50:48.552 INFO:teuthology.orchestra.run.smithi148.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-02-05T04:50:48.552 INFO:teuthology.orchestra.run.smithi148.stdout:Recommended packages: 2024-02-05T04:50:48.552 INFO:teuthology.orchestra.run.smithi148.stdout: btrfs-tools 2024-02-05T04:50:48.654 INFO:teuthology.orchestra.run.smithi148.stdout:The following NEW packages will be installed: 2024-02-05T04:50:48.654 INFO:teuthology.orchestra.run.smithi148.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-02-05T04:50:48.654 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-02-05T04:50:48.655 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-02-05T04:50:48.655 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-02-05T04:50:48.655 INFO:teuthology.orchestra.run.smithi148.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-02-05T04:50:48.655 INFO:teuthology.orchestra.run.smithi148.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-02-05T04:50:48.657 INFO:teuthology.orchestra.run.smithi148.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-02-05T04:50:48.658 INFO:teuthology.orchestra.run.smithi148.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-02-05T04:50:48.658 INFO:teuthology.orchestra.run.smithi148.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-02-05T04:50:48.658 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-02-05T04:50:48.658 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-02-05T04:50:48.658 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-02-05T04:50:48.658 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-02-05T04:50:48.658 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-02-05T04:50:48.658 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-02-05T04:50:48.658 INFO:teuthology.orchestra.run.smithi148.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-02-05T04:50:48.658 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-02-05T04:50:48.658 INFO:teuthology.orchestra.run.smithi148.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-02-05T04:50:48.658 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-02-05T04:50:48.658 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-02-05T04:50:48.659 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-02-05T04:50:48.659 INFO:teuthology.orchestra.run.smithi148.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-02-05T04:50:48.659 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-02-05T04:50:48.659 INFO:teuthology.orchestra.run.smithi148.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-02-05T04:50:48.659 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-02-05T04:50:48.659 INFO:teuthology.orchestra.run.smithi148.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-02-05T04:50:48.843 INFO:teuthology.orchestra.run.smithi148.stdout:0 upgraded, 118 newly installed, 0 to remove and 269 not upgraded. 2024-02-05T04:50:48.843 INFO:teuthology.orchestra.run.smithi148.stdout:Need to get 172 MB of archives. 2024-02-05T04:50:48.843 INFO:teuthology.orchestra.run.smithi148.stdout:After this operation, 713 MB of additional disk space will be used. 2024-02-05T04:50:48.843 INFO:teuthology.orchestra.run.smithi148.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-02-05T04:50:48.901 INFO:teuthology.orchestra.run.smithi148.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-658-gd4ed20cf-1jammy [3,530 kB] 2024-02-05T04:50:49.271 INFO:teuthology.orchestra.run.smithi148.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-02-05T04:50:49.283 INFO:teuthology.orchestra.run.smithi148.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-02-05T04:50:49.366 INFO:teuthology.orchestra.run.smithi148.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-02-05T04:50:49.596 INFO:teuthology.orchestra.run.smithi148.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-02-05T04:50:49.608 INFO:teuthology.orchestra.run.smithi148.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-02-05T04:50:49.637 INFO:teuthology.orchestra.run.smithi148.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-02-05T04:50:49.647 INFO:teuthology.orchestra.run.smithi148.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-02-05T04:50:49.647 INFO:teuthology.orchestra.run.smithi148.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-02-05T04:50:49.649 INFO:teuthology.orchestra.run.smithi148.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-02-05T04:50:49.667 INFO:teuthology.orchestra.run.smithi148.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-02-05T04:50:49.771 INFO:teuthology.orchestra.run.smithi148.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-658-gd4ed20cf-1jammy [3,207 kB] 2024-02-05T04:50:49.835 INFO:teuthology.orchestra.run.smithi148.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-02-05T04:50:50.104 INFO:teuthology.orchestra.run.smithi148.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-658-gd4ed20cf-1jammy [709 kB] 2024-02-05T04:50:50.168 INFO:teuthology.orchestra.run.smithi148.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-658-gd4ed20cf-1jammy [356 kB] 2024-02-05T04:50:50.201 INFO:teuthology.orchestra.run.smithi148.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-658-gd4ed20cf-1jammy [32.8 kB] 2024-02-05T04:50:50.202 INFO:teuthology.orchestra.run.smithi148.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-658-gd4ed20cf-1jammy [180 kB] 2024-02-05T04:50:50.209 INFO:teuthology.orchestra.run.smithi148.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-658-gd4ed20cf-1jammy [64.1 kB] 2024-02-05T04:50:50.211 INFO:teuthology.orchestra.run.smithi148.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-658-gd4ed20cf-1jammy [330 kB] 2024-02-05T04:50:50.240 INFO:teuthology.orchestra.run.smithi148.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-658-gd4ed20cf-1jammy [6,225 kB] 2024-02-05T04:50:50.244 INFO:teuthology.orchestra.run.smithi148.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-02-05T04:50:50.277 INFO:teuthology.orchestra.run.smithi148.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-02-05T04:50:50.280 INFO:teuthology.orchestra.run.smithi148.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-02-05T04:50:50.360 INFO:teuthology.orchestra.run.smithi148.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2024-02-05T04:50:50.411 INFO:teuthology.orchestra.run.smithi148.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-02-05T04:50:50.433 INFO:teuthology.orchestra.run.smithi148.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-02-05T04:50:50.439 INFO:teuthology.orchestra.run.smithi148.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-02-05T04:50:50.440 INFO:teuthology.orchestra.run.smithi148.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-02-05T04:50:50.454 INFO:teuthology.orchestra.run.smithi148.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-02-05T04:50:50.455 INFO:teuthology.orchestra.run.smithi148.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-02-05T04:50:50.604 INFO:teuthology.orchestra.run.smithi148.stdout:Get:32 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-658-gd4ed20cf-1jammy [112 kB] 2024-02-05T04:50:50.607 INFO:teuthology.orchestra.run.smithi148.stdout:Get:33 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-658-gd4ed20cf-1jammy [465 kB] 2024-02-05T04:50:50.617 INFO:teuthology.orchestra.run.smithi148.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-658-gd4ed20cf-1jammy [23.0 MB] 2024-02-05T04:50:50.618 INFO:teuthology.orchestra.run.smithi148.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-02-05T04:50:50.701 INFO:teuthology.orchestra.run.smithi148.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-02-05T04:50:50.730 INFO:teuthology.orchestra.run.smithi148.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-02-05T04:50:50.747 INFO:teuthology.orchestra.run.smithi148.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-02-05T04:50:50.791 INFO:teuthology.orchestra.run.smithi148.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-02-05T04:50:51.006 INFO:teuthology.orchestra.run.smithi148.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-02-05T04:50:51.021 INFO:teuthology.orchestra.run.smithi148.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-02-05T04:50:51.027 INFO:teuthology.orchestra.run.smithi148.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-02-05T04:50:51.033 INFO:teuthology.orchestra.run.smithi148.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-02-05T04:50:51.054 INFO:teuthology.orchestra.run.smithi148.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-02-05T04:50:51.058 INFO:teuthology.orchestra.run.smithi148.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-02-05T04:50:51.224 INFO:teuthology.orchestra.run.smithi148.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-02-05T04:50:51.487 INFO:teuthology.orchestra.run.smithi148.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-658-gd4ed20cf-1jammy [5,057 kB] 2024-02-05T04:50:51.568 INFO:teuthology.orchestra.run.smithi148.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-02-05T04:50:51.585 INFO:teuthology.orchestra.run.smithi148.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-658-gd4ed20cf-1jammy [240 kB] 2024-02-05T04:50:51.592 INFO:teuthology.orchestra.run.smithi148.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-658-gd4ed20cf-1jammy [124 kB] 2024-02-05T04:50:51.596 INFO:teuthology.orchestra.run.smithi148.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-658-gd4ed20cf-1jammy [1,519 kB] 2024-02-05T04:50:51.605 INFO:teuthology.orchestra.run.smithi148.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-02-05T04:50:51.616 INFO:teuthology.orchestra.run.smithi148.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-02-05T04:50:51.646 INFO:teuthology.orchestra.run.smithi148.stdout:Get:54 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-658-gd4ed20cf-1jammy [6,238 kB] 2024-02-05T04:50:51.820 INFO:teuthology.orchestra.run.smithi148.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-02-05T04:50:51.821 INFO:teuthology.orchestra.run.smithi148.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-02-05T04:50:51.827 INFO:teuthology.orchestra.run.smithi148.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-02-05T04:50:51.833 INFO:teuthology.orchestra.run.smithi148.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-02-05T04:50:51.848 INFO:teuthology.orchestra.run.smithi148.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-02-05T04:50:51.851 INFO:teuthology.orchestra.run.smithi148.stdout:Get:60 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-658-gd4ed20cf-1jammy [22.7 MB] 2024-02-05T04:50:51.869 INFO:teuthology.orchestra.run.smithi148.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-02-05T04:50:51.875 INFO:teuthology.orchestra.run.smithi148.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-02-05T04:50:52.051 INFO:teuthology.orchestra.run.smithi148.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-02-05T04:50:52.307 INFO:teuthology.orchestra.run.smithi148.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-02-05T04:50:52.485 INFO:teuthology.orchestra.run.smithi148.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-02-05T04:50:52.500 INFO:teuthology.orchestra.run.smithi148.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-02-05T04:50:52.520 INFO:teuthology.orchestra.run.smithi148.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-02-05T04:50:52.584 INFO:teuthology.orchestra.run.smithi148.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-02-05T04:50:52.696 INFO:teuthology.orchestra.run.smithi148.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-658-gd4ed20cf-1jammy [14.1 kB] 2024-02-05T04:50:52.697 INFO:teuthology.orchestra.run.smithi148.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-658-gd4ed20cf-1jammy [866 kB] 2024-02-05T04:50:52.716 INFO:teuthology.orchestra.run.smithi148.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-658-gd4ed20cf-1jammy [2,182 kB] 2024-02-05T04:50:52.753 INFO:teuthology.orchestra.run.smithi148.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-658-gd4ed20cf-1jammy [770 kB] 2024-02-05T04:50:52.763 INFO:teuthology.orchestra.run.smithi148.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-658-gd4ed20cf-1jammy [139 kB] 2024-02-05T04:50:52.764 INFO:teuthology.orchestra.run.smithi148.stdout:Get:74 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-658-gd4ed20cf-1jammy [1,876 kB] 2024-02-05T04:50:52.812 INFO:teuthology.orchestra.run.smithi148.stdout:Get:75 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-658-gd4ed20cf-1jammy [8,625 kB] 2024-02-05T04:50:52.874 INFO:teuthology.orchestra.run.smithi148.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-02-05T04:50:52.876 INFO:teuthology.orchestra.run.smithi148.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-02-05T04:50:52.878 INFO:teuthology.orchestra.run.smithi148.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-02-05T04:50:52.879 INFO:teuthology.orchestra.run.smithi148.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-02-05T04:50:52.881 INFO:teuthology.orchestra.run.smithi148.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-02-05T04:50:53.096 INFO:teuthology.orchestra.run.smithi148.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-658-gd4ed20cf-1jammy [14.2 kB] 2024-02-05T04:50:53.097 INFO:teuthology.orchestra.run.smithi148.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-658-gd4ed20cf-1jammy [48.4 MB] 2024-02-05T04:50:53.110 INFO:teuthology.orchestra.run.smithi148.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-02-05T04:50:53.547 INFO:teuthology.orchestra.run.smithi148.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-02-05T04:50:53.553 INFO:teuthology.orchestra.run.smithi148.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-02-05T04:50:53.843 INFO:teuthology.orchestra.run.smithi148.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-02-05T04:50:53.843 INFO:teuthology.orchestra.run.smithi148.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-02-05T04:50:53.845 INFO:teuthology.orchestra.run.smithi148.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-02-05T04:50:53.847 INFO:teuthology.orchestra.run.smithi148.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-02-05T04:50:53.849 INFO:teuthology.orchestra.run.smithi148.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-02-05T04:50:53.851 INFO:teuthology.orchestra.run.smithi148.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-02-05T04:50:53.852 INFO:teuthology.orchestra.run.smithi148.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-02-05T04:50:53.854 INFO:teuthology.orchestra.run.smithi148.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-02-05T04:50:54.037 INFO:teuthology.orchestra.run.smithi148.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-02-05T04:50:54.453 INFO:teuthology.orchestra.run.smithi148.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-02-05T04:50:54.501 INFO:teuthology.orchestra.run.smithi148.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-02-05T04:50:54.535 INFO:teuthology.orchestra.run.smithi148.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-02-05T04:50:54.633 INFO:teuthology.orchestra.run.smithi148.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-02-05T04:50:54.673 INFO:teuthology.orchestra.run.smithi148.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-02-05T04:50:54.756 INFO:teuthology.orchestra.run.smithi148.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-02-05T04:50:54.757 INFO:teuthology.orchestra.run.smithi148.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-02-05T04:50:54.788 INFO:teuthology.orchestra.run.smithi148.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-02-05T04:50:54.789 INFO:teuthology.orchestra.run.smithi148.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-02-05T04:50:54.833 INFO:teuthology.orchestra.run.smithi148.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-02-05T04:50:54.922 INFO:teuthology.orchestra.run.smithi148.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-658-gd4ed20cf-1jammy [121 kB] 2024-02-05T04:50:54.925 INFO:teuthology.orchestra.run.smithi148.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-658-gd4ed20cf-1jammy [38.4 kB] 2024-02-05T04:50:54.926 INFO:teuthology.orchestra.run.smithi148.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-658-gd4ed20cf-1jammy [12.3 MB] 2024-02-05T04:50:54.998 INFO:teuthology.orchestra.run.smithi148.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-02-05T04:50:55.235 INFO:teuthology.orchestra.run.smithi148.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-nitzan-max-pglog-size/d4ed20cfb238ef2c47d64ca4533eca00b8968b98/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-658-gd4ed20cf-1jammy [91.7 kB] 2024-02-05T04:50:55.270 INFO:teuthology.orchestra.run.smithi148.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-02-05T04:50:55.288 INFO:teuthology.orchestra.run.smithi148.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-02-05T04:50:55.443 INFO:teuthology.orchestra.run.smithi148.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-02-05T04:50:55.472 INFO:teuthology.orchestra.run.smithi148.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-02-05T04:50:55.630 INFO:teuthology.orchestra.run.smithi148.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-02-05T04:50:55.633 INFO:teuthology.orchestra.run.smithi148.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-02-05T04:50:55.635 INFO:teuthology.orchestra.run.smithi148.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-02-05T04:50:55.658 INFO:teuthology.orchestra.run.smithi148.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-02-05T04:50:55.662 INFO:teuthology.orchestra.run.smithi148.stdout:Get:118 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-02-05T04:50:56.326 INFO:teuthology.orchestra.run.smithi148.stdout:Fetched 172 MB in 7s (24.2 MB/s) 2024-02-05T04:50:56.461 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-02-05T04:50:56.559 INFO:teuthology.orchestra.run.smithi148.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165989 files and directories currently installed.) 2024-02-05T04:50:56.564 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-02-05T04:50:56.659 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-05T04:50:56.946 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-02-05T04:50:56.970 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-02-05T04:50:57.004 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-05T04:50:57.307 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-02-05T04:50:57.322 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-02-05T04:50:57.365 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-05T04:50:57.819 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-02-05T04:50:57.845 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-05T04:50:57.894 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-05T04:50:58.255 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-02-05T04:50:58.282 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-05T04:50:58.313 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-05T04:50:58.633 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-02-05T04:50:58.659 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-02-05T04:50:58.700 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-05T04:50:59.053 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-02-05T04:50:59.079 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-02-05T04:50:59.119 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-05T04:50:59.439 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package librados2. 2024-02-05T04:50:59.465 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../007-librados2_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:50:59.506 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking librados2 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:50:59.900 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package librbd1. 2024-02-05T04:50:59.923 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../008-librbd1_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:50:59.958 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking librbd1 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:00.369 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libcephfs2. 2024-02-05T04:51:00.396 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:00.436 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libcephfs2 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:00.697 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-rados. 2024-02-05T04:51:00.723 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../010-python3-rados_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:00.763 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-rados (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:01.099 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-02-05T04:51:01.126 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-05T04:51:01.166 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-ceph-argparse (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:01.402 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-cephfs. 2024-02-05T04:51:01.417 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:01.451 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-cephfs (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:01.713 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-ceph-common. 2024-02-05T04:51:01.739 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-05T04:51:01.771 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-ceph-common (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:02.057 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-wcwidth. 2024-02-05T04:51:02.084 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-02-05T04:51:02.124 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-05T04:51:02.443 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-prettytable. 2024-02-05T04:51:02.469 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-02-05T04:51:02.509 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-02-05T04:51:03.004 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-rbd. 2024-02-05T04:51:03.019 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:03.054 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-rbd (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:03.323 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-02-05T04:51:03.339 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-02-05T04:51:03.373 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-05T04:51:03.701 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-02-05T04:51:03.727 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-02-05T04:51:03.767 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-02-05T04:51:04.070 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-02-05T04:51:04.086 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-02-05T04:51:04.129 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-05T04:51:04.440 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package lua5.1. 2024-02-05T04:51:04.455 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-02-05T04:51:04.490 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-02-05T04:51:04.801 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package lua-any. 2024-02-05T04:51:04.816 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-02-05T04:51:04.860 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking lua-any (27ubuntu1) ... 2024-02-05T04:51:05.097 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package zip. 2024-02-05T04:51:05.124 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-02-05T04:51:05.164 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking zip (3.0-12build2) ... 2024-02-05T04:51:05.442 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package unzip. 2024-02-05T04:51:05.468 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.1_amd64.deb ... 2024-02-05T04:51:05.500 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2024-02-05T04:51:05.846 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package luarocks. 2024-02-05T04:51:05.872 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-02-05T04:51:05.912 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-02-05T04:51:06.287 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package librgw2. 2024-02-05T04:51:06.313 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../025-librgw2_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:06.353 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking librgw2 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:06.872 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-rgw. 2024-02-05T04:51:06.895 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:06.939 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-rgw (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:07.242 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package liboath0:amd64. 2024-02-05T04:51:07.269 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-02-05T04:51:07.308 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-02-05T04:51:07.611 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libradosstriper1. 2024-02-05T04:51:07.637 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:07.677 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libradosstriper1 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:08.123 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-common. 2024-02-05T04:51:08.149 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../029-ceph-common_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:08.189 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-common (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:09.453 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-base. 2024-02-05T04:51:09.479 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../030-ceph-base_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:09.529 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-base (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:10.099 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-02-05T04:51:10.125 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-02-05T04:51:10.165 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-02-05T04:51:10.443 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-cheroot. 2024-02-05T04:51:10.469 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-02-05T04:51:10.501 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-05T04:51:10.779 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-02-05T04:51:10.794 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-02-05T04:51:10.829 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-02-05T04:51:11.115 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jaraco.text. 2024-02-05T04:51:11.141 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-02-05T04:51:11.181 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-02-05T04:51:11.484 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-02-05T04:51:11.510 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-02-05T04:51:11.550 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-02-05T04:51:11.853 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-tempora. 2024-02-05T04:51:11.880 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-02-05T04:51:11.920 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-02-05T04:51:12.156 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-portend. 2024-02-05T04:51:12.183 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-02-05T04:51:12.214 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-02-05T04:51:12.517 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-02-05T04:51:12.544 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-02-05T04:51:12.584 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-02-05T04:51:12.887 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-cherrypy3. 2024-02-05T04:51:12.913 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-02-05T04:51:12.953 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-02-05T04:51:13.482 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-natsort. 2024-02-05T04:51:13.509 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-02-05T04:51:13.548 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-02-05T04:51:13.860 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-logutils. 2024-02-05T04:51:13.887 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-02-05T04:51:13.926 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-02-05T04:51:14.204 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-markupsafe. 2024-02-05T04:51:14.231 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-02-05T04:51:14.271 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-02-05T04:51:14.549 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-mako. 2024-02-05T04:51:14.576 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-02-05T04:51:14.615 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-05T04:51:14.910 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-simplegeneric. 2024-02-05T04:51:14.926 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-02-05T04:51:14.960 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-02-05T04:51:15.230 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-singledispatch. 2024-02-05T04:51:15.256 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-02-05T04:51:15.296 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-02-05T04:51:15.640 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-webob. 2024-02-05T04:51:15.667 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-02-05T04:51:15.707 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-02-05T04:51:16.052 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-waitress. 2024-02-05T04:51:16.078 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-02-05T04:51:16.121 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-05T04:51:16.404 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-tempita. 2024-02-05T04:51:16.431 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-02-05T04:51:16.471 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-02-05T04:51:16.732 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-paste. 2024-02-05T04:51:16.759 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-02-05T04:51:16.798 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-02-05T04:51:17.127 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-02-05T04:51:17.142 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-02-05T04:51:17.177 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-02-05T04:51:17.413 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pastedeploy. 2024-02-05T04:51:17.439 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-02-05T04:51:17.479 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-02-05T04:51:17.791 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-webtest. 2024-02-05T04:51:17.818 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-02-05T04:51:17.857 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-02-05T04:51:18.168 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pecan. 2024-02-05T04:51:18.196 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-02-05T04:51:18.235 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-02-05T04:51:18.607 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-certifi. 2024-02-05T04:51:18.634 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-02-05T04:51:18.673 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-02-05T04:51:18.993 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-idna. 2024-02-05T04:51:19.020 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-02-05T04:51:19.051 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-idna (3.3-1) ... 2024-02-05T04:51:19.329 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-urllib3. 2024-02-05T04:51:19.356 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-02-05T04:51:19.395 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-05T04:51:19.682 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-requests. 2024-02-05T04:51:19.709 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-02-05T04:51:19.748 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-05T04:51:20.059 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-werkzeug. 2024-02-05T04:51:20.088 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-02-05T04:51:20.126 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-05T04:51:20.421 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-02-05T04:51:20.448 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-05T04:51:20.487 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-modules-core (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:20.884 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-02-05T04:51:20.911 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:20.950 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:21.253 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-bcrypt. 2024-02-05T04:51:21.280 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-02-05T04:51:21.319 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-02-05T04:51:21.564 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-openssl. 2024-02-05T04:51:21.591 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-02-05T04:51:21.622 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-02-05T04:51:21.967 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr. 2024-02-05T04:51:21.994 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:22.109 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:23.195 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mon. 2024-02-05T04:51:23.222 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:23.261 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mon (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:23.973 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-osd. 2024-02-05T04:51:24.000 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:24.040 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-osd (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:25.076 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph. 2024-02-05T04:51:25.092 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../066-ceph_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:25.134 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:25.454 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-fuse. 2024-02-05T04:51:25.481 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:25.520 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-fuse (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:25.857 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mds. 2024-02-05T04:51:25.884 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:25.915 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mds (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:26.326 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package cephadm. 2024-02-05T04:51:26.354 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../069-cephadm_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:26.393 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking cephadm (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:26.704 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-asyncssh. 2024-02-05T04:51:26.731 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-02-05T04:51:26.770 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-02-05T04:51:27.065 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python-babel-localedata. 2024-02-05T04:51:27.080 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-02-05T04:51:27.123 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-05T04:51:28.120 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-babel. 2024-02-05T04:51:28.147 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-02-05T04:51:28.186 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-02-05T04:51:28.506 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-jinja2. 2024-02-05T04:51:28.533 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-02-05T04:51:28.572 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-05T04:51:28.917 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-02-05T04:51:28.941 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-05T04:51:28.983 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-cephadm (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:29.328 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-repoze.lru. 2024-02-05T04:51:29.355 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-02-05T04:51:29.395 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-02-05T04:51:29.664 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-routes. 2024-02-05T04:51:29.691 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-02-05T04:51:29.747 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-02-05T04:51:30.026 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-02-05T04:51:30.041 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-05T04:51:30.075 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-dashboard (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:30.495 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-02-05T04:51:30.523 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-02-05T04:51:30.562 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-05T04:51:31.041 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-joblib. 2024-02-05T04:51:31.067 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-02-05T04:51:31.107 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-02-05T04:51:31.460 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-02-05T04:51:31.487 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-02-05T04:51:31.527 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-02-05T04:51:31.829 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-sklearn. 2024-02-05T04:51:31.856 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-02-05T04:51:31.896 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-05T04:51:32.638 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-02-05T04:51:32.665 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-05T04:51:32.781 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:33.760 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-cachetools. 2024-02-05T04:51:33.787 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-02-05T04:51:33.981 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-02-05T04:51:34.287 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pyasn1. 2024-02-05T04:51:34.314 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-02-05T04:51:34.354 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-02-05T04:51:34.665 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-02-05T04:51:34.692 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-02-05T04:51:34.732 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-02-05T04:51:35.043 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-rsa. 2024-02-05T04:51:35.070 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-02-05T04:51:35.110 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-rsa (4.8-1) ... 2024-02-05T04:51:35.380 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-google-auth. 2024-02-05T04:51:35.407 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-02-05T04:51:35.438 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-02-05T04:51:35.725 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-02-05T04:51:35.740 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-02-05T04:51:35.774 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-05T04:51:36.061 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-websocket. 2024-02-05T04:51:36.088 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-02-05T04:51:36.127 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-02-05T04:51:36.447 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-kubernetes. 2024-02-05T04:51:36.474 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-02-05T04:51:36.513 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-05T04:51:37.203 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-02-05T04:51:37.230 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-05T04:51:37.270 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:37.548 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libonig5:amd64. 2024-02-05T04:51:37.574 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-02-05T04:51:37.613 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-05T04:51:37.958 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libjq1:amd64. 2024-02-05T04:51:37.985 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-02-05T04:51:38.024 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-05T04:51:38.285 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package jq. 2024-02-05T04:51:38.301 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-02-05T04:51:38.343 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-02-05T04:51:38.588 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package socat. 2024-02-05T04:51:38.606 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-02-05T04:51:38.638 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-02-05T04:51:38.908 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package xmlstarlet. 2024-02-05T04:51:38.925 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-02-05T04:51:38.966 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-02-05T04:51:39.262 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-test. 2024-02-05T04:51:39.277 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../097-ceph-test_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:39.311 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-test (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:41.349 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package ceph-volume. 2024-02-05T04:51:41.377 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-658-gd4ed20cf-1jammy_all.deb ... 2024-02-05T04:51:41.407 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking ceph-volume (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:41.703 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package libcephfs-dev. 2024-02-05T04:51:41.718 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:41.761 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking libcephfs-dev (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:42.047 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package lua-socket:amd64. 2024-02-05T04:51:42.074 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-02-05T04:51:42.114 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-05T04:51:42.437 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package lua-sec:amd64. 2024-02-05T04:51:42.464 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-02-05T04:51:42.503 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-02-05T04:51:42.849 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package nvme-cli. 2024-02-05T04:51:42.876 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-02-05T04:51:42.915 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-02-05T04:51:43.312 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package pkg-config. 2024-02-05T04:51:43.339 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-02-05T04:51:43.378 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-02-05T04:51:43.598 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-02-05T04:51:43.614 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-02-05T04:51:43.656 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-02-05T04:51:44.235 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-iniconfig. 2024-02-05T04:51:44.262 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-02-05T04:51:44.292 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-02-05T04:51:44.595 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pastescript. 2024-02-05T04:51:44.623 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-02-05T04:51:44.661 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-02-05T04:51:44.965 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pluggy. 2024-02-05T04:51:44.993 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-02-05T04:51:45.031 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-02-05T04:51:45.351 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-psutil. 2024-02-05T04:51:45.378 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-02-05T04:51:45.426 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-02-05T04:51:45.746 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-py. 2024-02-05T04:51:45.773 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-02-05T04:51:45.812 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-py (1.10.0-1) ... 2024-02-05T04:51:46.141 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pygments. 2024-02-05T04:51:46.168 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-02-05T04:51:46.207 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-02-05T04:51:46.636 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pyinotify. 2024-02-05T04:51:46.652 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-02-05T04:51:46.686 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-02-05T04:51:46.989 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-toml. 2024-02-05T04:51:47.016 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-02-05T04:51:47.055 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-02-05T04:51:47.341 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-pytest. 2024-02-05T04:51:47.368 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-02-05T04:51:47.407 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-02-05T04:51:47.761 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package python3-simplejson. 2024-02-05T04:51:47.789 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-02-05T04:51:47.827 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-02-05T04:51:48.097 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-02-05T04:51:48.113 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-02-05T04:51:48.155 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-02-05T04:51:48.809 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package radosgw. 2024-02-05T04:51:48.837 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../116-radosgw_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:48.875 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking radosgw (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:49.912 INFO:teuthology.orchestra.run.smithi148.stdout:Selecting previously unselected package rbd-fuse. 2024-02-05T04:51:49.928 INFO:teuthology.orchestra.run.smithi148.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-658-gd4ed20cf-1jammy_amd64.deb ... 2024-02-05T04:51:49.970 INFO:teuthology.orchestra.run.smithi148.stdout:Unpacking rbd-fuse (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:50.257 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-02-05T04:51:50.500 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-02-05T04:51:50.617 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-02-05T04:51:50.889 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-02-05T04:51:51.264 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-02-05T04:51:51.676 INFO:teuthology.orchestra.run.smithi148.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-02-05T04:51:51.752 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up cephadm (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:52.296 INFO:teuthology.orchestra.run.smithi148.stdout:Adding system user cephadm....done 2024-02-05T04:51:52.468 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-02-05T04:51:53.053 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-02-05T04:51:53.313 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-02-05T04:51:53.439 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-02-05T04:51:53.707 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-02-05T04:51:53.958 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-02-05T04:51:54.067 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-py (1.10.0-1) ... 2024-02-05T04:51:54.377 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-02-05T04:51:54.738 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-02-05T04:51:55.014 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-02-05T04:51:55.290 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2024-02-05T04:51:55.541 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-02-05T04:51:55.792 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-02-05T04:51:56.060 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-02-05T04:51:56.461 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-ceph-argparse (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:51:56.712 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-02-05T04:51:56.813 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-02-05T04:51:56.922 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-02-05T04:51:57.048 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-02-05T04:51:57.299 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-02-05T04:51:57.661 INFO:teuthology.orchestra.run.smithi148.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-02-05T04:51:57.697 INFO:teuthology.orchestra.run.smithi148.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-02-05T04:51:57.761 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-02-05T04:51:57.887 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-02-05T04:51:58.247 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-02-05T04:51:58.356 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-02-05T04:51:58.657 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-02-05T04:51:58.900 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-02-05T04:51:59.201 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up zip (3.0-12build2) ... 2024-02-05T04:51:59.335 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-02-05T04:52:00.317 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-02-05T04:52:00.613 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-02-05T04:52:00.739 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-02-05T04:52:00.848 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-02-05T04:52:01.132 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-02-05T04:52:01.517 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-02-05T04:52:01.777 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-02-05T04:52:02.153 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-02-05T04:52:02.438 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-02-05T04:52:02.764 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-02-05T04:52:02.998 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-02-05T04:52:03.124 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-ceph-common (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:03.442 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-idna (3.3-1) ... 2024-02-05T04:52:03.768 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-02-05T04:52:04.966 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-02-05T04:52:05.217 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-05T04:52:05.352 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-toml (0.10.2-1) ... 2024-02-05T04:52:05.636 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-02-05T04:52:05.745 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-02-05T04:52:05.846 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-02-05T04:52:06.139 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-02-05T04:52:06.375 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-02-05T04:52:06.625 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-05T04:52:06.759 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-02-05T04:52:07.069 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-02-05T04:52:07.329 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-02-05T04:52:07.580 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-02-05T04:52:07.822 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-02-05T04:52:08.065 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-02-05T04:52:08.324 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-02-05T04:52:08.434 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-02-05T04:52:08.743 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-02-05T04:52:08.869 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-02-05T04:52:09.179 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-02-05T04:52:09.496 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-02-05T04:52:09.764 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-02-05T04:52:09.865 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-02-05T04:52:09.966 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-02-05T04:52:10.039 INFO:teuthology.orchestra.run.smithi148.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-02-05T04:52:10.285 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-02-05T04:52:10.410 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-02-05T04:52:10.795 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-02-05T04:52:11.055 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up lua-any (27ubuntu1) ... 2024-02-05T04:52:11.181 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-portend (3.0.0-1) ... 2024-02-05T04:52:11.432 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-02-05T04:52:11.558 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-02-05T04:52:11.875 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-02-05T04:52:12.193 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-02-05T04:52:12.504 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-02-05T04:52:12.629 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-rsa (4.8-1) ... 2024-02-05T04:52:12.905 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-02-05T04:52:13.181 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-02-05T04:52:13.517 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-02-05T04:52:13.902 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-02-05T04:52:14.179 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-02-05T04:52:14.489 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-02-05T04:52:14.590 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up librados2 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:14.692 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libsqlite3-mod-ceph (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:14.818 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-02-05T04:52:15.154 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libcephfs2 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:15.280 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libradosstriper1 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:15.406 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-02-05T04:52:15.666 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up librbd1 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:15.767 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-modules-core (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:16.060 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-fuse (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:16.288 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-05T04:52:16.288 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-02-05T04:52:16.802 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up libcephfs-dev (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:16.904 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-rados (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:17.004 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up librgw2 (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:17.130 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-02-05T04:52:18.300 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-rbd (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:18.427 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up rbd-fuse (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:18.528 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-rgw (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:18.629 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up python3-cephfs (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:18.738 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-common (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:19.264 INFO:teuthology.orchestra.run.smithi148.stdout:Adding group ceph....done 2024-02-05T04:52:19.631 INFO:teuthology.orchestra.run.smithi148.stdout:Adding system user ceph....done 2024-02-05T04:52:19.751 INFO:teuthology.orchestra.run.smithi148.stdout:Setting system user ceph properties....done 2024-02-05T04:52:19.769 INFO:teuthology.orchestra.run.smithi148.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-02-05T04:52:19.901 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-02-05T04:52:20.183 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-02-05T04:52:20.639 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-test (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:20.765 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up radosgw (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:21.223 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-05T04:52:21.223 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-02-05T04:52:21.716 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-base (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:22.053 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-02-05T04:52:22.510 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mds (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-05T04:52:22.738 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-02-05T04:52:23.212 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:23.431 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-05T04:52:23.431 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-02-05T04:52:23.905 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-osd (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:24.185 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-05T04:52:24.185 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-02-05T04:52:24.681 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-k8sevents (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:24.782 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:24.916 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mon (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:25.139 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-05T04:52:25.139 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-02-05T04:52:25.635 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-cephadm (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:25.777 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:25.903 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-mgr-dashboard (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:26.029 INFO:teuthology.orchestra.run.smithi148.stdout:Setting up ceph-volume (19.0.0-658-gd4ed20cf-1jammy) ... 2024-02-05T04:52:26.382 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-02-05T04:52:26.483 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-02-05T04:52:26.745 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-02-05T04:52:27.779 INFO:teuthology.orchestra.run.smithi148.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-02-05T04:52:28.408 INFO:teuthology.orchestra.run.smithi148.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-02-05T04:52:28.413 DEBUG:teuthology.parallel:result is None 2024-02-05T04:52:28.414 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-05T04:52:28.650 DEBUG:teuthology.orchestra.run.smithi148:> dpkg-query -W -f '${Version}' ceph 2024-02-05T04:52:28.669 INFO:teuthology.orchestra.run.smithi148.stdout:19.0.0-658-gd4ed20cf-1jammy 2024-02-05T04:52:28.670 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-658-gd4ed20cf-1jammy 2024-02-05T04:52:28.670 INFO:teuthology.task.install:The correct ceph version 19.0.0-658-gd4ed20cf-1jammy is installed. 2024-02-05T04:52:28.675 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-05T04:52:28.675 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-05T04:52:28.675 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-05T04:52:28.726 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-05T04:52:28.726 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-05T04:52:28.727 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/daemon-helper 2024-02-05T04:52:28.782 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-05T04:52:28.838 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-05T04:52:28.838 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-05T04:52:28.838 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-05T04:52:28.898 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-05T04:52:28.954 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-05T04:52:28.954 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-05T04:52:28.954 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/stdin-killer 2024-02-05T04:52:29.009 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-05T04:52:29.061 INFO:teuthology.run_tasks:Running task workunit... 2024-02-05T04:52:29.076 INFO:tasks.workunit:Pulling workunits from ref d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-05T04:52:29.076 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-05T04:52:29.076 INFO:tasks.workunit:timeout=3h 2024-02-05T04:52:29.076 INFO:tasks.workunit:cleanup=True 2024-02-05T04:52:29.076 DEBUG:teuthology.orchestra.run.smithi148:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-05T04:52:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:52:29.107 INFO:teuthology.orchestra.run.smithi148.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-02-05T04:52:29.107 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-02-05T04:52:29.154 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-02-05T04:52:29.154 DEBUG:teuthology.orchestra.run.smithi148:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-02-05T04:52:29.203 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d4ed20cfb238ef2c47d64ca4533eca00b8968b98 2024-02-05T04:52:29.252 INFO:tasks.workunit.client.0.smithi148.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-05T04:53:36.000 INFO:tasks.workunit.client.0.smithi148.stderr:Note: switching to 'd4ed20cfb238ef2c47d64ca4533eca00b8968b98'. 2024-02-05T04:53:36.000 INFO:tasks.workunit.client.0.smithi148.stderr: 2024-02-05T04:53:36.000 INFO:tasks.workunit.client.0.smithi148.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-05T04:53:36.000 INFO:tasks.workunit.client.0.smithi148.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-05T04:53:36.001 INFO:tasks.workunit.client.0.smithi148.stderr:state without impacting any branches by switching back to a branch. 2024-02-05T04:53:36.001 INFO:tasks.workunit.client.0.smithi148.stderr: 2024-02-05T04:53:36.001 INFO:tasks.workunit.client.0.smithi148.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-05T04:53:36.001 INFO:tasks.workunit.client.0.smithi148.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-05T04:53:36.001 INFO:tasks.workunit.client.0.smithi148.stderr: 2024-02-05T04:53:36.001 INFO:tasks.workunit.client.0.smithi148.stderr: git switch -c 2024-02-05T04:53:36.001 INFO:tasks.workunit.client.0.smithi148.stderr: 2024-02-05T04:53:36.001 INFO:tasks.workunit.client.0.smithi148.stderr:Or undo this operation with: 2024-02-05T04:53:36.001 INFO:tasks.workunit.client.0.smithi148.stderr: 2024-02-05T04:53:36.001 INFO:tasks.workunit.client.0.smithi148.stderr: git switch - 2024-02-05T04:53:36.001 INFO:tasks.workunit.client.0.smithi148.stderr: 2024-02-05T04:53:36.001 INFO:tasks.workunit.client.0.smithi148.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-05T04:53:36.001 INFO:tasks.workunit.client.0.smithi148.stderr: 2024-02-05T04:53:36.001 INFO:tasks.workunit.client.0.smithi148.stderr:HEAD is now at d4ed20cfb23 ceph_dencoder: memory leaks when loading dencoder plugins 2024-02-05T04:53:36.003 DEBUG:teuthology.orchestra.run.smithi148:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-05T04:53:36.051 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-02-05T04:53:36.051 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-05T04:53:36.097 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-02-05T04:53:36.098 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-02-05T04:53:36.098 DEBUG:teuthology.orchestra.run.smithi148:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-02-05T04:53:36.170 INFO:tasks.workunit.client.0.smithi148.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-05T04:53:36.175 INFO:tasks.workunit.client.0.smithi148.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-05T04:53:36.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-05T04:53:36.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-05T04:53:36.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-05T04:53:36.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-05T04:53:36.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-05T04:53:36.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-05T04:53:36.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-05T04:53:36.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-05T04:53:36.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-mapper 2024-02-05T04:53:36.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-02-05T04:53:36.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-02-05T04:53:36.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-02-05T04:53:36.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-02-05T04:53:36.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-02-05T04:53:36.176 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-02-05T04:53:36.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=42955096-cab3-4e29-ab89-af002f474456 --auth-supported=none ' 2024-02-05T04:53:36.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-02-05T04:53:36.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-02-05T04:53:36.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-02-05T04:53:36.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-05T04:53:36.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-02-05T04:53:36.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-02-05T04:53:36.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-02-05T04:53:36.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2024-02-05T04:53:36.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2024-02-05T04:53:36.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-02-05T04:53:36.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:53:36.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-02-05T04:53:36.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:53:36.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:53:36.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:53:36.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:53:36.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:53:36.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:53:36.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:53:36.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:53:36.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:53:36.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:53:36.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:53:36.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:53:36.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:53:36.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:53:36.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:53:36.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:53:36.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:53:36.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:53:36.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-02-05T04:53:36.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:53:36.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:53:36.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20103 2024-02-05T04:53:36.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20103 2024-02-05T04:53:36.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:53:36.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:53:36.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2024-02-05T04:53:36.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T04:53:36.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:53:36.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20103 2024-02-05T04:53:36.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20103 2024-02-05T04:53:36.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T04:53:36.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-05T04:53:36.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-05T04:53:36.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T04:53:36.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-02-05T04:53:36.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-02-05T04:53:36.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-02-05T04:53:36.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-02-05T04:53:36.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-02-05T04:53:36.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-02-05T04:53:36.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-02-05T04:53:36.195 INFO:tasks.workunit.client.0.smithi148.stdout:Dir: td/osd-mapper 2024-02-05T04:53:36.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-02-05T04:53:36.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-02-05T04:53:36.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2024-02-05T04:53:36.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-02-05T04:53:36.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-02-05T04:53:36.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-02-05T04:53:36.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-02-05T04:53:36.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-02-05T04:53:36.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-02-05T04:53:36.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-02-05T04:53:36.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-02-05T04:53:36.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-02-05T04:53:37.706 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 44def9ba-32d8-4313-b826-77c7fca29a17 2024-02-05T04:53:37.996 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T04:53:38.084 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:53:38.081+0000 7f92ac1bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:38.116 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:53:38.113+0000 7f92ac1bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:38.117 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:53:38.117+0000 7f92ac1bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:38.118 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:53:38.117+0000 7f92ac1bd640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-02-05T04:53:38.118 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:53:38.117+0000 7f92ac1bd640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-02-05T04:53:38.118 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:53:38.117+0000 7f92ac1bd640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-02-05T04:53:38.165 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:53:38.161+0000 7f92ac1bd640 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-02-05T04:53:44.113 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T04:53:44.321 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T04:53:44.355 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:53:44.353+0000 7fd1467b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:44.389 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:53:44.389+0000 7fd1467b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:44.390 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:53:44.389+0000 7fd1467b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:44.537 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T04:53:44.940 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:53:44.937+0000 7fd1467b1640 -1 Falling back to public interface 2024-02-05T04:53:45.755 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T04:53:46.934 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T04:53:47.384 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:53:47.365+0000 7fd1467b1640 -1 osd.0 0 log_to_monitors true 2024-02-05T04:53:48.241 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T04:53:49.563 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T04:53:50.870 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T04:53:52.266 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T04:53:53.581 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T04:53:54.897 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T04:53:56.227 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T04:53:57.532 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T04:53:58.838 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T04:54:00.457 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T04:54:01.805 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T04:54:03.545 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T04:54:05.455 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T04:54:06.808 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T04:54:08.526 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T04:54:08.837 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3984853459,v1:127.0.0.1:6801/3984853459] [v2:127.0.0.1:6802/3984853459,v1:127.0.0.1:6803/3984853459] exists,up 44def9ba-32d8-4313-b826-77c7fca29a17 2024-02-05T04:54:08.843 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 a81824e4-d160-4bde-a938-519581181562 2024-02-05T04:54:09.206 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T04:54:09.263 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:09.261+0000 7fe938d95640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:09.281 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:09.281+0000 7fe938d95640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:09.282 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:09.281+0000 7fe938d95640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:09.283 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:09.281+0000 7fe938d95640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-02-05T04:54:09.283 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:09.281+0000 7fe938d95640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-02-05T04:54:09.283 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:09.281+0000 7fe938d95640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-02-05T04:54:09.348 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:09.345+0000 7fe938d95640 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-02-05T04:54:17.353 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T04:54:17.754 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T04:54:17.781 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:17.780+0000 7fd51c16c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:17.784 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:17.780+0000 7fd51c16c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:17.785 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:17.784+0000 7fd51c16c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:18.071 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T04:54:18.319 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:18.316+0000 7fd51c16c640 -1 Falling back to public interface 2024-02-05T04:54:19.400 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T04:54:20.706 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T04:54:20.817 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:20.812+0000 7fd51c16c640 -1 osd.1 0 log_to_monitors true 2024-02-05T04:54:22.028 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T04:54:23.359 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T04:54:24.796 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T04:54:26.095 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T04:54:27.453 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T04:54:28.783 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T04:54:30.095 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T04:54:31.434 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T04:54:32.818 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T04:54:34.117 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T04:54:35.676 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T04:54:37.383 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T04:54:39.337 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T04:54:41.053 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T04:54:42.495 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T04:54:43.809 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T04:54:45.456 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T04:54:45.768 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2987985148,v1:127.0.0.1:6811/2987985148] [v2:127.0.0.1:6812/2987985148,v1:127.0.0.1:6813/2987985148] exists,up a81824e4-d160-4bde-a938-519581181562 2024-02-05T04:54:45.774 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 0d8c914c-2a20-4f16-844b-9c10f945c6be 2024-02-05T04:54:46.179 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T04:54:46.237 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:46.231+0000 7f5001545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:46.266 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:46.263+0000 7f5001545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:46.267 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:46.263+0000 7f5001545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:46.268 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:46.263+0000 7f5001545640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-02-05T04:54:46.268 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:46.263+0000 7f5001545640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-02-05T04:54:46.268 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:46.263+0000 7f5001545640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-02-05T04:54:46.334 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:46.331+0000 7f5001545640 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-02-05T04:54:52.116 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T04:54:52.504 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T04:54:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:52.535+0000 7fbe69c52640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:52.562 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:52.559+0000 7fbe69c52640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:52.563 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:52.559+0000 7fbe69c52640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:52.830 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T04:54:53.116 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:53.111+0000 7fbe69c52640 -1 Falling back to public interface 2024-02-05T04:54:54.150 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T04:54:55.470 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T04:54:55.582 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:54:55.574+0000 7fbe69c52640 -1 osd.2 0 log_to_monitors true 2024-02-05T04:54:56.782 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T04:54:58.097 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T04:54:59.434 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T04:55:00.800 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T04:55:02.222 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T04:55:03.534 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T04:55:04.840 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T04:55:06.222 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T04:55:07.586 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T04:55:09.006 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T04:55:10.455 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T04:55:12.149 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T04:55:13.530 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T04:55:15.063 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T04:55:16.781 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T04:55:18.158 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T04:55:18.496 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/814071486,v1:127.0.0.1:6819/814071486] [v2:127.0.0.1:6820/814071486,v1:127.0.0.1:6821/814071486] exists,up 0d8c914c-2a20-4f16-844b-9c10f945c6be 2024-02-05T04:55:18.962 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'test' created 2024-02-05T04:55:20.951 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836499 2024-02-05T04:55:22.557 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672972 2024-02-05T04:55:22.904 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509447 2024-02-05T04:55:25.632 INFO:tasks.workunit.client.0.smithi148.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-02-05T04:55:25.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-02-05T04:55:25.782 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T04:55:25.782 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-02-05T04:55:25.782 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T04:55:25.795 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T04:55:25.796 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-02-05T04:55:25.796 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T04:55:25.808 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2: { 2024-02-05T04:55:25.808 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-02-05T04:55:25.808 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T04:55:25.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-02-05T04:55:25.947 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T04:55:25.947 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-02-05T04:55:25.947 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T04:55:25.959 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T04:55:25.959 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-02-05T04:55:25.959 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T04:55:25.971 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2: { 2024-02-05T04:55:25.971 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-02-05T04:55:25.971 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T04:55:25.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-02-05T04:55:25.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-02-05T04:55:25.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-02-05T04:55:25.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-02-05T04:55:25.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-02-05T04:55:25.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-02-05T04:55:25.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-02-05T04:55:26.354 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap01 2024-02-05T04:55:26.552 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap02 2024-02-05T04:55:26.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-02-05T04:55:26.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-02-05T04:55:26.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-02-05T04:55:26.967 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap13 2024-02-05T04:55:26.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-02-05T04:55:26.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-02-05T04:55:26.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-02-05T04:55:27.369 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap24 2024-02-05T04:55:27.575 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap25 2024-02-05T04:55:27.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 18507 2024-02-05T04:55:27.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-02-05T04:55:27.906 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-02-05T04:55:27.906 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-02-05T04:55:28.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-02-05T04:55:28.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-02-05T04:55:28.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-02-05T04:55:28.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-02-05T04:55:28.547 INFO:tasks.workunit.client.0.smithi148.stdout:pgid is 1.3 (primary: osd.1) 2024-02-05T04:55:28.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-02-05T04:55:28.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-02-05T04:55:28.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:294: set_query_debug: local pgid=1.3 2024-02-05T04:55:28.548 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: ceph pg dump pgs_brief 2024-02-05T04:55:28.548 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-02-05T04:55:28.862 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs_brief 2024-02-05T04:55:28.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: local prim_osd=1 2024-02-05T04:55:28.879 INFO:tasks.workunit.client.0.smithi148.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-02-05T04:55:28.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-02-05T04:55:28.880 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: get_asok_path osd.1 2024-02-05T04:55:28.880 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T04:55:28.880 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T04:55:28.880 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:55:28.880 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:55:28.880 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20103 2024-02-05T04:55:28.880 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20103/ceph-osd.1.asok 2024-02-05T04:55:28.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: CEPH_ARGS= 2024-02-05T04:55:28.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.20103/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-02-05T04:55:28.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:55:28.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-02-05T04:55:29.026 INFO:tasks.workunit.client.0.smithi148.stdout:{"success":true}{ 2024-02-05T04:55:29.027 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "objxxx", 2024-02-05T04:55:29.027 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T04:55:29.027 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:55:29.027 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 2, 2024-02-05T04:55:29.027 INFO:tasks.workunit.client.0.smithi148.stdout: "snapshots": [ 2024-02-05T04:55:29.027 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:55:29.027 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 1, 2024-02-05T04:55:29.027 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "snap01" 2024-02-05T04:55:29.027 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T04:55:29.027 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:55:29.028 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 2, 2024-02-05T04:55:29.028 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "snap02" 2024-02-05T04:55:29.028 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T04:55:29.028 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T04:55:29.028 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 5, 2024-02-05T04:55:29.028 INFO:tasks.workunit.client.0.smithi148.stdout: "overlaps": [] 2024-02-05T04:55:29.028 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T04:55:29.028 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:55:29.028 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 3, 2024-02-05T04:55:29.028 INFO:tasks.workunit.client.0.smithi148.stdout: "snapshots": [ 2024-02-05T04:55:29.028 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:55:29.028 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 3, 2024-02-05T04:55:29.028 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "snap13" 2024-02-05T04:55:29.029 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T04:55:29.029 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T04:55:29.029 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 3, 2024-02-05T04:55:29.029 INFO:tasks.workunit.client.0.smithi148.stdout: "overlaps": [] 2024-02-05T04:55:29.029 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T04:55:29.029 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:55:29.029 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 5, 2024-02-05T04:55:29.029 INFO:tasks.workunit.client.0.smithi148.stdout: "snapshots": [ 2024-02-05T04:55:29.029 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:55:29.029 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 4, 2024-02-05T04:55:29.029 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "snap24" 2024-02-05T04:55:29.029 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T04:55:29.029 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:55:29.029 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 5, 2024-02-05T04:55:29.030 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "snap25" 2024-02-05T04:55:29.030 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T04:55:29.030 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T04:55:29.030 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 5, 2024-02-05T04:55:29.030 INFO:tasks.workunit.client.0.smithi148.stdout: "overlaps": [] 2024-02-05T04:55:29.030 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T04:55:29.030 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:55:29.030 INFO:tasks.workunit.client.0.smithi148.stdout: "id": "head", 2024-02-05T04:55:29.030 INFO:tasks.workunit.client.0.smithi148.stdout: "snapshots": [], 2024-02-05T04:55:29.030 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 6 2024-02-05T04:55:29.030 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T04:55:29.030 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T04:55:29.030 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T04:55:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-02-05T04:55:29.143 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T04:55:29.144 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": true, 2024-02-05T04:55:29.144 INFO:tasks.workunit.client.0.smithi148.stdout: "must": true, 2024-02-05T04:55:29.144 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "0.000000" 2024-02-05T04:55:29.144 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T04:55:29.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-02-05T04:55:32.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-05T04:55:32.457 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:55:32.458 INFO:tasks.workunit.client.0.smithi148.stdout:1.3 4 0 0 0 0 19 0 0 7 0 7 active+clean 2024-02-05T04:55:29.953759+0000 25'7 25:38 [1,2,0] 1 [1,2,0] 1 25'7 2024-02-05T04:55:29.953575+0000 25'7 2024-02-05T04:55:29.953575+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:55:29.953575+0000 4 0 2024-02-05T04:55:32.458 INFO:tasks.workunit.client.0.smithi148.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:55:20.070195+0000 0'0 25:23 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:55:18.910100+0000 0 0 2024-02-05T04:55:32.458 INFO:tasks.workunit.client.0.smithi148.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:55:20.070548+0000 0'0 25:23 [2,0,1] 2 [2,0,1] 2 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:55:18.910100+0000 0 0 2024-02-05T04:55:32.458 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:55:20.070653+0000 0'0 25:24 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:55:18.910100+0000 0 0 2024-02-05T04:55:32.458 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:32.458 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:55:32.458 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T04:55:32.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-02-05T04:55:32.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-05T04:55:32.764 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:55:32.764 INFO:tasks.workunit.client.0.smithi148.stdout:1.3 4 0 0 0 0 19 0 0 7 0 7 active+clean 2024-02-05T04:55:29.953759+0000 25'7 25:38 [1,2,0] 1 [1,2,0] 1 25'7 2024-02-05T04:55:29.953575+0000 25'7 2024-02-05T04:55:29.953575+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:55:29.953575+0000 4 0 2024-02-05T04:55:32.764 INFO:tasks.workunit.client.0.smithi148.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:55:20.070195+0000 0'0 25:23 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:55:18.910100+0000 0 0 2024-02-05T04:55:32.764 INFO:tasks.workunit.client.0.smithi148.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:55:20.070548+0000 0'0 25:23 [2,0,1] 2 [2,0,1] 2 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:55:18.910100+0000 0 0 2024-02-05T04:55:32.764 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:55:20.070653+0000 0'0 25:24 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:55:18.910100+0000 0 0 2024-02-05T04:55:32.764 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:32.765 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:55:32.765 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T04:55:32.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-02-05T04:55:33.192 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T04:55:33.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-02-05T04:55:33.786 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T04:55:33.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-02-05T04:55:38.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-02-05T04:55:38.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-02-05T04:55:38.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:55:38.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:55:38.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:55:38.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:55:38.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:55:39.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:55:39.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:55:39.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-02-05T04:55:39.150 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.145+0000 7f493042b940 1 bdev(0x5617a8d7a000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-05T04:55:39.151 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.149+0000 7f493042b940 1 bdev(0x5617a8d7a000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:39.151 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.149+0000 7f493042b940 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-05T04:55:39.151 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.149+0000 7f493042b940 1 bdev(0x5617a8d7a700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-05T04:55:39.151 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.149+0000 7f493042b940 1 bdev(0x5617a8d7a700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:39.151 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.149+0000 7f493042b940 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-02-05T04:55:39.151 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.149+0000 7f493042b940 1 bluefs mount 2024-02-05T04:55:39.168 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.165+0000 7f493042b940 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:55:39.193 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.189+0000 7f493042b940 1 bluefs mount shared_bdev_used = 0 2024-02-05T04:55:39.193 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.189+0000 7f493042b940 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:55:39.197 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.193+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:55:39.197 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.193+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:55:39.197 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.193+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:55:39.197 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.193+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:55:39.197 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.193+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:55:39.197 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.193+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:55:39.197 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.193+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:55:39.197 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.193+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:55:39.198 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.193+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:55:39.198 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.193+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:55:39.198 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.193+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:55:39.198 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.193+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:55:39.247 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.245+0000 7f493042b940 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:55:39.247 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.245+0000 7f493042b940 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-02-05T04:55:39.247 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.245+0000 7f493042b940 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-02-05T04:55:39.247 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.245+0000 7f493042b940 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-05T04:55:39.247 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.245+0000 7f493042b940 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-02-05T04:55:39.247 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.245+0000 7f493042b940 1 freelist init 2024-02-05T04:55:39.247 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.245+0000 7f493042b940 1 freelist _read_cfg 2024-02-05T04:55:39.248 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.245+0000 7f493042b940 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 22 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc2000, fragmentation 8.01088e-07 2024-02-05T04:55:39.248 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.245+0000 7f493042b940 1 bluefs umount 2024-02-05T04:55:39.248 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.245+0000 7f493042b940 1 bdev(0x5617a8d7a700 td/osd-mapper/0/block) close 2024-02-05T04:55:39.431 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.429+0000 7f493042b940 1 bdev(0x5617a8d7a700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-05T04:55:39.431 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.429+0000 7f493042b940 1 bdev(0x5617a8d7a700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:39.431 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.429+0000 7f493042b940 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-02-05T04:55:39.431 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.429+0000 7f493042b940 1 bluefs mount 2024-02-05T04:55:39.431 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.429+0000 7f493042b940 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:55:39.438 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.433+0000 7f493042b940 1 bluefs mount shared_bdev_used = 27459584 2024-02-05T04:55:39.438 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.433+0000 7f493042b940 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:55:39.441 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.437+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:55:39.441 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.437+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:55:39.441 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.437+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:55:39.441 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.437+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:55:39.441 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.437+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:55:39.441 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.437+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:55:39.441 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.437+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:55:39.441 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.437+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:55:39.441 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.437+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:55:39.441 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.437+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:55:39.441 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.437+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:55:39.442 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:39.437+0000 7f493042b940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:55:40.330 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.325+0000 7f493042b940 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:55:40.331 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-05T04:55:40.333 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 0b 05 58 02 00 00 42 95 50 96 ca b3 4e 29 ab 89 |..X...B.P...N)..| 2024-02-05T04:55:40.333 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 af 00 2f 47 44 56 00 00 00 00 1b 00 00 00 00 00 |../GDV..........| 2024-02-05T04:55:40.333 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-05T04:55:40.333 INFO:tasks.workunit.client.0.smithi148.stdout:* 2024-02-05T04:55:40.333 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-05T04:55:40.333 INFO:tasks.workunit.client.0.smithi148.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-05T04:55:40.333 INFO:tasks.workunit.client.0.smithi148.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-05T04:55:40.333 INFO:tasks.workunit.client.0.smithi148.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-05T04:55:40.334 INFO:tasks.workunit.client.0.smithi148.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-05T04:55:40.334 INFO:tasks.workunit.client.0.smithi148.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-05T04:55:40.334 INFO:tasks.workunit.client.0.smithi148.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-05T04:55:40.334 INFO:tasks.workunit.client.0.smithi148.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-05T04:55:40.334 INFO:tasks.workunit.client.0.smithi148.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-05T04:55:40.334 INFO:tasks.workunit.client.0.smithi148.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-05T04:55:40.334 INFO:tasks.workunit.client.0.smithi148.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-05T04:55:40.334 INFO:tasks.workunit.client.0.smithi148.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-05T04:55:40.334 INFO:tasks.workunit.client.0.smithi148.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-05T04:55:40.334 INFO:tasks.workunit.client.0.smithi148.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-05T04:55:40.334 INFO:tasks.workunit.client.0.smithi148.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-05T04:55:40.334 INFO:tasks.workunit.client.0.smithi148.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-05T04:55:40.334 INFO:tasks.workunit.client.0.smithi148.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-05T04:55:40.334 INFO:tasks.workunit.client.0.smithi148.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-05T04:55:40.335 INFO:tasks.workunit.client.0.smithi148.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-05T04:55:40.335 INFO:tasks.workunit.client.0.smithi148.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-05T04:55:40.335 INFO:tasks.workunit.client.0.smithi148.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-05T04:55:40.335 INFO:tasks.workunit.client.0.smithi148.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-05T04:55:40.335 INFO:tasks.workunit.client.0.smithi148.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-05T04:55:40.335 INFO:tasks.workunit.client.0.smithi148.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-05T04:55:40.335 INFO:tasks.workunit.client.0.smithi148.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-05T04:55:40.335 INFO:tasks.workunit.client.0.smithi148.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-05T04:55:40.335 INFO:tasks.workunit.client.0.smithi148.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-05T04:55:40.335 INFO:tasks.workunit.client.0.smithi148.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-05T04:55:40.335 INFO:tasks.workunit.client.0.smithi148.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-05T04:55:40.335 INFO:tasks.workunit.client.0.smithi148.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-05T04:55:40.335 INFO:tasks.workunit.client.0.smithi148.stdout:00000220 72 65 1b 00 00 00 05 00 00 00 44 de f9 ba 32 d8 |re........D...2.| 2024-02-05T04:55:40.335 INFO:tasks.workunit.client.0.smithi148.stdout:00000230 43 13 b8 26 77 c7 fc a2 9a 17 00 00 00 00 00 00 |C..&w...........| 2024-02-05T04:55:40.336 INFO:tasks.workunit.client.0.smithi148.stdout:00000240 00 00 1b 00 00 00 f0 69 c0 65 a4 30 91 11 01 00 |.......i.e.0....| 2024-02-05T04:55:40.336 INFO:tasks.workunit.client.0.smithi148.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1b 00 00 00 |..............| 2024-02-05T04:55:40.336 INFO:tasks.workunit.client.0.smithi148.stdout:0000025e 2024-02-05T04:55:40.336 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:40.336 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-05T04:55:40.336 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:40.336 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:40.336 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:55:40.336 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:40.336 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:40.336 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-05T04:55:40.336 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:55:40.336 INFO:tasks.workunit.client.0.smithi148.stdout:00000047 2024-02-05T04:55:40.337 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:40.337 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:40.337 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-05T04:55:40.337 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:40.337 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:40.337 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-05T04:55:40.337 INFO:tasks.workunit.client.0.smithi148.stdout:0000003f 2024-02-05T04:55:40.337 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:40.337 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:40.337 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:55:40.337 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:40.337 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:40.337 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-05T04:55:40.337 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:55:40.338 INFO:tasks.workunit.client.0.smithi148.stdout:00000047 2024-02-05T04:55:40.338 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:40.338 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:40.338 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:40.338 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:40.338 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:40.338 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:40.338 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:40.338 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:40.338 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:40.338 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:40.338 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:40.338 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:40.339 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:40.339 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:40.339 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:40.339 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:40.339 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:40.339 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:40.339 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:40.339 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:40.339 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:40.339 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:40.339 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:40.339 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:40.339 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:40.340 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:40.340 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:40.340 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:40.340 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:40.340 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:40.340 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:40.340 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:40.340 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:40.340 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:40.340 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:40.340 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:40.340 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-05T04:55:40.340 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:40.340 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-05T04:55:40.341 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:40.341 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.329+0000 7f493042b940 1 bluefs umount 2024-02-05T04:55:40.341 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.329+0000 7f493042b940 1 bdev(0x5617a8d7a700 td/osd-mapper/0/block) close 2024-02-05T04:55:40.595 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.593+0000 7f493042b940 1 freelist shutdown 2024-02-05T04:55:40.595 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.593+0000 7f493042b940 1 bdev(0x5617a8d7a000 td/osd-mapper/0/block) close 2024-02-05T04:55:40.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:55:40.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-02-05T04:55:40.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-02-05T04:55:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.717+0000 7fe6d7201940 1 bdev(0x55ace9412000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-05T04:55:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.717+0000 7fe6d7201940 1 bdev(0x55ace9412000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.717+0000 7fe6d7201940 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-05T04:55:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.717+0000 7fe6d7201940 1 bdev(0x55ace9412700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-05T04:55:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.717+0000 7fe6d7201940 1 bdev(0x55ace9412700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.717+0000 7fe6d7201940 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-02-05T04:55:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.717+0000 7fe6d7201940 1 bluefs mount 2024-02-05T04:55:40.727 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.725+0000 7fe6d7201940 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:55:40.735 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.733+0000 7fe6d7201940 1 bluefs mount shared_bdev_used = 0 2024-02-05T04:55:40.735 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.733+0000 7fe6d7201940 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:55:40.739 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.733+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:55:40.739 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.733+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:55:40.739 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.733+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:55:40.739 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.733+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:55:40.739 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.737+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:55:40.739 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.737+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:55:40.739 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.737+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:55:40.739 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.737+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:55:40.740 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.737+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:55:40.740 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.737+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:55:40.740 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.737+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:55:40.740 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.737+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:55:40.787 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.781+0000 7fe6d7201940 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:55:40.787 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.781+0000 7fe6d7201940 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-02-05T04:55:40.787 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.781+0000 7fe6d7201940 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-02-05T04:55:40.787 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.781+0000 7fe6d7201940 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-05T04:55:40.787 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.781+0000 7fe6d7201940 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-02-05T04:55:40.787 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.785+0000 7fe6d7201940 1 freelist init 2024-02-05T04:55:40.787 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.785+0000 7fe6d7201940 1 freelist _read_cfg 2024-02-05T04:55:40.788 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.785+0000 7fe6d7201940 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc2000, fragmentation 4.95912e-07 2024-02-05T04:55:40.788 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.785+0000 7fe6d7201940 1 bluefs umount 2024-02-05T04:55:40.789 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:40.785+0000 7fe6d7201940 1 bdev(0x55ace9412700 td/osd-mapper/2/block) close 2024-02-05T04:55:41.007 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.005+0000 7fe6d7201940 1 bdev(0x55ace9412700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-05T04:55:41.007 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.005+0000 7fe6d7201940 1 bdev(0x55ace9412700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:41.007 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.005+0000 7fe6d7201940 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-02-05T04:55:41.007 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.005+0000 7fe6d7201940 1 bluefs mount 2024-02-05T04:55:41.019 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.017+0000 7fe6d7201940 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:55:41.028 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.025+0000 7fe6d7201940 1 bluefs mount shared_bdev_used = 27459584 2024-02-05T04:55:41.028 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.025+0000 7fe6d7201940 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:55:41.033 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.029+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:55:41.033 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.029+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:55:41.033 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.029+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:55:41.033 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.029+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:55:41.033 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.029+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:55:41.033 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.029+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:55:41.033 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.029+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:55:41.033 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.029+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:55:41.033 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.029+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:55:41.033 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.029+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:55:41.034 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.029+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:55:41.034 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.029+0000 7fe6d7201940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:55:41.859 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.857+0000 7fe6d7201940 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:55:41.861 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.857+0000 7fe6d7201940 1 bluefs umount 2024-02-05T04:55:41.861 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:41.857+0000 7fe6d7201940 1 bdev(0x55ace9412700 td/osd-mapper/2/block) close 2024-02-05T04:55:42.091 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:42.089+0000 7fe6d7201940 1 freelist shutdown 2024-02-05T04:55:42.091 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:42.089+0000 7fe6d7201940 1 bdev(0x55ace9412000 td/osd-mapper/2/block) close 2024-02-05T04:55:42.259 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:42.259 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:42.259 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:42.259 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:42.259 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:42.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:55:42.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-02-05T04:55:42.260 INFO:tasks.workunit.client.0.smithi148.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-02-05T04:55:42.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:55:42.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-02-05T04:55:42.846 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-05T04:55:42.846 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 0b 05 58 02 00 00 42 95 50 96 ca b3 4e 29 ab 89 |..X...B.P...N)..| 2024-02-05T04:55:42.846 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 af 00 2f 47 44 56 02 00 00 00 1b 00 00 00 00 00 |../GDV..........| 2024-02-05T04:55:42.846 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-05T04:55:42.846 INFO:tasks.workunit.client.0.smithi148.stdout:* 2024-02-05T04:55:42.847 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-05T04:55:42.847 INFO:tasks.workunit.client.0.smithi148.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-05T04:55:42.847 INFO:tasks.workunit.client.0.smithi148.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-05T04:55:42.847 INFO:tasks.workunit.client.0.smithi148.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-05T04:55:42.847 INFO:tasks.workunit.client.0.smithi148.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-05T04:55:42.847 INFO:tasks.workunit.client.0.smithi148.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-05T04:55:42.847 INFO:tasks.workunit.client.0.smithi148.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-05T04:55:42.847 INFO:tasks.workunit.client.0.smithi148.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-05T04:55:42.847 INFO:tasks.workunit.client.0.smithi148.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-05T04:55:42.847 INFO:tasks.workunit.client.0.smithi148.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-05T04:55:42.847 INFO:tasks.workunit.client.0.smithi148.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-05T04:55:42.847 INFO:tasks.workunit.client.0.smithi148.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-05T04:55:42.847 INFO:tasks.workunit.client.0.smithi148.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-05T04:55:42.847 INFO:tasks.workunit.client.0.smithi148.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-05T04:55:42.848 INFO:tasks.workunit.client.0.smithi148.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-05T04:55:42.848 INFO:tasks.workunit.client.0.smithi148.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-05T04:55:42.848 INFO:tasks.workunit.client.0.smithi148.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-05T04:55:42.848 INFO:tasks.workunit.client.0.smithi148.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-05T04:55:42.848 INFO:tasks.workunit.client.0.smithi148.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-05T04:55:42.848 INFO:tasks.workunit.client.0.smithi148.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-05T04:55:42.848 INFO:tasks.workunit.client.0.smithi148.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-05T04:55:42.848 INFO:tasks.workunit.client.0.smithi148.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-05T04:55:42.848 INFO:tasks.workunit.client.0.smithi148.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-05T04:55:42.848 INFO:tasks.workunit.client.0.smithi148.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-05T04:55:42.848 INFO:tasks.workunit.client.0.smithi148.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-05T04:55:42.848 INFO:tasks.workunit.client.0.smithi148.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-05T04:55:42.848 INFO:tasks.workunit.client.0.smithi148.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-05T04:55:42.849 INFO:tasks.workunit.client.0.smithi148.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-05T04:55:42.849 INFO:tasks.workunit.client.0.smithi148.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-05T04:55:42.849 INFO:tasks.workunit.client.0.smithi148.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-05T04:55:42.849 INFO:tasks.workunit.client.0.smithi148.stdout:00000220 72 65 1b 00 00 00 0f 00 00 00 0d 8c 91 4c 2a 20 |re...........L* | 2024-02-05T04:55:42.849 INFO:tasks.workunit.client.0.smithi148.stdout:00000230 4f 16 84 4b 9c 10 f9 45 c6 be 00 00 00 00 00 00 |O..K...E........| 2024-02-05T04:55:42.849 INFO:tasks.workunit.client.0.smithi148.stdout:00000240 00 00 1b 00 00 00 35 6a c0 65 70 7b 42 39 01 00 |......5j.ep{B9..| 2024-02-05T04:55:42.849 INFO:tasks.workunit.client.0.smithi148.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1b 00 00 00 |..............| 2024-02-05T04:55:42.849 INFO:tasks.workunit.client.0.smithi148.stdout:0000025e 2024-02-05T04:55:42.849 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:42.849 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-05T04:55:42.849 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:42.849 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:42.849 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:55:42.849 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:42.850 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:42.850 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-05T04:55:42.850 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:55:42.850 INFO:tasks.workunit.client.0.smithi148.stdout:00000047 2024-02-05T04:55:42.850 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:42.850 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:42.850 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-05T04:55:42.850 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:42.850 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:42.850 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-05T04:55:42.850 INFO:tasks.workunit.client.0.smithi148.stdout:0000003f 2024-02-05T04:55:42.850 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:42.850 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:42.850 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:55:42.851 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:42.851 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:42.851 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-05T04:55:42.851 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:55:42.851 INFO:tasks.workunit.client.0.smithi148.stdout:00000047 2024-02-05T04:55:42.851 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:42.851 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:42.851 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:42.851 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:42.851 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:42.851 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:42.851 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:42.851 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:42.852 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:42.852 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:42.852 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:42.852 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:42.852 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:42.852 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:42.852 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:42.852 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:42.852 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:42.852 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:42.852 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:42.852 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:42.852 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:42.853 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:42.853 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:42.853 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:42.853 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:42.853 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:42.853 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:42.853 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:42.853 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:42.853 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:42.853 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:42.853 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:42.854 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:42.854 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:42.854 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:42.854 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:42.854 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-05T04:55:42.854 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:42.854 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-05T04:55:42.854 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:43.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-05T04:55:43.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-02-05T04:55:43.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-05T04:55:43.331 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-02-05T04:55:44.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-02-05T04:55:44.916 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-02-05T04:55:44.918 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-02-05T04:55:44.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:55:44.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-02-05T04:55:44.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-02-05T04:55:44.919 INFO:tasks.workunit.client.0.smithi148.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-02-05T04:55:44.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-02-05T04:55:44.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-02-05T04:55:44.949 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.945+0000 7fcf5bebc940 1 bdev(0x560f2f352000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-05T04:55:44.949 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.945+0000 7fcf5bebc940 1 bdev(0x560f2f352000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:44.949 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.945+0000 7fcf5bebc940 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-05T04:55:44.950 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.945+0000 7fcf5bebc940 1 bdev(0x560f2f352700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-05T04:55:44.950 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.945+0000 7fcf5bebc940 1 bdev(0x560f2f352700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:44.950 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.945+0000 7fcf5bebc940 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-02-05T04:55:44.950 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.945+0000 7fcf5bebc940 1 bluefs mount 2024-02-05T04:55:44.959 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.957+0000 7fcf5bebc940 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:55:44.968 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.965+0000 7fcf5bebc940 1 bluefs mount shared_bdev_used = 0 2024-02-05T04:55:44.968 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.965+0000 7fcf5bebc940 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:55:44.972 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.969+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:55:44.972 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.969+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:55:44.972 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.969+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:55:44.972 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.969+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:55:44.972 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.969+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:55:44.972 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.969+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:55:44.972 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.969+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:55:44.972 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.969+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:55:44.972 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.969+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:55:44.972 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.969+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:55:44.972 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.969+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:55:44.973 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.969+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:55:44.977 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.973+0000 7fcf5bebc940 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:55:44.978 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.973+0000 7fcf5bebc940 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-02-05T04:55:44.978 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.973+0000 7fcf5bebc940 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-02-05T04:55:44.978 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.973+0000 7fcf5bebc940 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-05T04:55:44.978 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.973+0000 7fcf5bebc940 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-02-05T04:55:44.978 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.973+0000 7fcf5bebc940 1 freelist init 2024-02-05T04:55:44.978 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.973+0000 7fcf5bebc940 1 freelist _read_cfg 2024-02-05T04:55:44.979 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.977+0000 7fcf5bebc940 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 22 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc2000, fragmentation 8.01088e-07 2024-02-05T04:55:44.979 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.977+0000 7fcf5bebc940 1 bluefs umount 2024-02-05T04:55:44.979 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:44.977+0000 7fcf5bebc940 1 bdev(0x560f2f352700 td/osd-mapper/0/block) close 2024-02-05T04:55:45.231 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.229+0000 7fcf5bebc940 1 bdev(0x560f2f352700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-05T04:55:45.231 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.229+0000 7fcf5bebc940 1 bdev(0x560f2f352700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:45.231 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.229+0000 7fcf5bebc940 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-02-05T04:55:45.231 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.229+0000 7fcf5bebc940 1 bluefs mount 2024-02-05T04:55:45.242 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.237+0000 7fcf5bebc940 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:55:45.252 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.249+0000 7fcf5bebc940 1 bluefs mount shared_bdev_used = 9568256 2024-02-05T04:55:45.252 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.249+0000 7fcf5bebc940 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:55:45.255 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.253+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:55:45.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.253+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:55:45.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.253+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:55:45.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.253+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:55:45.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.253+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:55:45.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.253+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:55:45.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.253+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:55:45.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.253+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:55:45.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.253+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:55:45.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.253+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:55:45.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.253+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:55:45.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.253+0000 7fcf5bebc940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:55:45.517 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.513+0000 7fcf5bebc940 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:55:45.519 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-05T04:55:45.519 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 0b 05 58 02 00 00 42 95 50 96 ca b3 4e 29 ab 89 |..X...B.P...N)..| 2024-02-05T04:55:45.519 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 af 00 2f 47 44 56 00 00 00 00 1b 00 00 00 00 00 |../GDV..........| 2024-02-05T04:55:45.519 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-05T04:55:45.519 INFO:tasks.workunit.client.0.smithi148.stdout:* 2024-02-05T04:55:45.520 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-05T04:55:45.520 INFO:tasks.workunit.client.0.smithi148.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-05T04:55:45.520 INFO:tasks.workunit.client.0.smithi148.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-05T04:55:45.520 INFO:tasks.workunit.client.0.smithi148.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-05T04:55:45.520 INFO:tasks.workunit.client.0.smithi148.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-05T04:55:45.521 INFO:tasks.workunit.client.0.smithi148.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-05T04:55:45.521 INFO:tasks.workunit.client.0.smithi148.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-05T04:55:45.521 INFO:tasks.workunit.client.0.smithi148.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-05T04:55:45.521 INFO:tasks.workunit.client.0.smithi148.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-05T04:55:45.521 INFO:tasks.workunit.client.0.smithi148.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-05T04:55:45.521 INFO:tasks.workunit.client.0.smithi148.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-05T04:55:45.522 INFO:tasks.workunit.client.0.smithi148.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-05T04:55:45.522 INFO:tasks.workunit.client.0.smithi148.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-05T04:55:45.522 INFO:tasks.workunit.client.0.smithi148.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-05T04:55:45.522 INFO:tasks.workunit.client.0.smithi148.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-05T04:55:45.522 INFO:tasks.workunit.client.0.smithi148.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-05T04:55:45.522 INFO:tasks.workunit.client.0.smithi148.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-05T04:55:45.523 INFO:tasks.workunit.client.0.smithi148.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-05T04:55:45.523 INFO:tasks.workunit.client.0.smithi148.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-05T04:55:45.523 INFO:tasks.workunit.client.0.smithi148.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-05T04:55:45.523 INFO:tasks.workunit.client.0.smithi148.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-05T04:55:45.523 INFO:tasks.workunit.client.0.smithi148.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-05T04:55:45.523 INFO:tasks.workunit.client.0.smithi148.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-05T04:55:45.523 INFO:tasks.workunit.client.0.smithi148.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-05T04:55:45.524 INFO:tasks.workunit.client.0.smithi148.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-05T04:55:45.524 INFO:tasks.workunit.client.0.smithi148.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-05T04:55:45.524 INFO:tasks.workunit.client.0.smithi148.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-05T04:55:45.524 INFO:tasks.workunit.client.0.smithi148.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-05T04:55:45.524 INFO:tasks.workunit.client.0.smithi148.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-05T04:55:45.524 INFO:tasks.workunit.client.0.smithi148.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-05T04:55:45.525 INFO:tasks.workunit.client.0.smithi148.stdout:00000220 72 65 1b 00 00 00 05 00 00 00 44 de f9 ba 32 d8 |re........D...2.| 2024-02-05T04:55:45.525 INFO:tasks.workunit.client.0.smithi148.stdout:00000230 43 13 b8 26 77 c7 fc a2 9a 17 00 00 00 00 00 00 |C..&w...........| 2024-02-05T04:55:45.525 INFO:tasks.workunit.client.0.smithi148.stdout:00000240 00 00 1b 00 00 00 f0 69 c0 65 a4 30 91 11 01 00 |.......i.e.0....| 2024-02-05T04:55:45.525 INFO:tasks.workunit.client.0.smithi148.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1b 00 00 00 |..............| 2024-02-05T04:55:45.525 INFO:tasks.workunit.client.0.smithi148.stdout:0000025e 2024-02-05T04:55:45.525 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:45.525 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-05T04:55:45.526 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:45.526 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:45.526 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:55:45.526 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:45.526 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:45.526 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-05T04:55:45.526 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:55:45.526 INFO:tasks.workunit.client.0.smithi148.stdout:00000047 2024-02-05T04:55:45.527 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:45.527 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:45.527 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-05T04:55:45.527 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:45.527 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:45.527 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-05T04:55:45.527 INFO:tasks.workunit.client.0.smithi148.stdout:0000003f 2024-02-05T04:55:45.527 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:45.527 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:45.528 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:55:45.528 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:45.528 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:45.528 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-05T04:55:45.528 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:55:45.528 INFO:tasks.workunit.client.0.smithi148.stdout:00000047 2024-02-05T04:55:45.528 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:45.529 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:45.529 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:45.529 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:45.529 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:45.529 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:45.529 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:45.529 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:45.530 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:45.530 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:45.530 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:45.530 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:45.530 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:45.530 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:45.531 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:45.531 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:45.531 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:45.531 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:45.531 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:45.531 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:45.531 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:45.531 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:45.531 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:45.531 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:45.531 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:45.531 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:45.531 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:45.531 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:45.532 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:45.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:45.532 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:45.532 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:45.532 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:45.532 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:45.532 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:45.532 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:45.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-05T04:55:45.532 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:45.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-05T04:55:45.532 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:45.533 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.517+0000 7fcf5bebc940 1 bluefs umount 2024-02-05T04:55:45.533 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.517+0000 7fcf5bebc940 1 bdev(0x560f2f352700 td/osd-mapper/0/block) close 2024-02-05T04:55:45.779 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.777+0000 7fcf5bebc940 1 freelist shutdown 2024-02-05T04:55:45.779 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:45.777+0000 7fcf5bebc940 1 bdev(0x560f2f352000 td/osd-mapper/0/block) close 2024-02-05T04:55:45.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-02-05T04:55:45.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-02-05T04:55:45.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-05T04:55:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:55:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-02-05T04:55:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-02-05T04:55:47.055 INFO:tasks.workunit.client.0.smithi148.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:47.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-02-05T04:55:47.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.l2MjixoOSS_the_val 2024-02-05T04:55:47.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:55:47.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.l2MjixoOSS_the_val' 2024-02-05T04:55:47.058 INFO:tasks.workunit.client.0.smithi148.stdout:Value dumped in: /tmp/tmp.l2MjixoOSS_the_val 2024-02-05T04:55:47.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.l2MjixoOSS_the_val 2024-02-05T04:55:47.586 INFO:tasks.workunit.client.0.smithi148.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-02-05T04:55:48.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:55:48.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.l2MjixoOSS_the_val 2024-02-05T04:55:48.137 INFO:tasks.workunit.client.0.smithi148.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-02-05T04:55:48.137 INFO:tasks.workunit.client.0.smithi148.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-02-05T04:55:48.137 INFO:tasks.workunit.client.0.smithi148.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-02-05T04:55:48.137 INFO:tasks.workunit.client.0.smithi148.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-02-05T04:55:48.137 INFO:tasks.workunit.client.0.smithi148.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-02-05T04:55:48.137 INFO:tasks.workunit.client.0.smithi148.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-02-05T04:55:48.137 INFO:tasks.workunit.client.0.smithi148.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-02-05T04:55:48.137 INFO:tasks.workunit.client.0.smithi148.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-02-05T04:55:48.137 INFO:tasks.workunit.client.0.smithi148.stdout:0000073 2024-02-05T04:55:48.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-02-05T04:55:48.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:49.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.l2MjixoOSS_the_val 2024-02-05T04:55:50.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:55:50.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:55:50.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-02-05T04:55:50.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-02-05T04:55:50.529 INFO:tasks.workunit.client.0.smithi148.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-02-05T04:55:50.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-02-05T04:55:50.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-05T04:55:50.558 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.553+0000 7f8dbea41940 1 bdev(0x55ff64b60000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-02-05T04:55:50.559 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.553+0000 7f8dbea41940 1 bdev(0x55ff64b60000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:50.559 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.557+0000 7f8dbea41940 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-05T04:55:50.559 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.557+0000 7f8dbea41940 1 bdev(0x55ff64b60700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-02-05T04:55:50.559 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.557+0000 7f8dbea41940 1 bdev(0x55ff64b60700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:50.559 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.557+0000 7f8dbea41940 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-02-05T04:55:50.559 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.557+0000 7f8dbea41940 1 bluefs mount 2024-02-05T04:55:50.567 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.565+0000 7f8dbea41940 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:55:50.576 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.573+0000 7f8dbea41940 1 bluefs mount shared_bdev_used = 0 2024-02-05T04:55:50.576 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.573+0000 7f8dbea41940 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:55:50.580 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.577+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:55:50.580 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.577+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:55:50.580 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.577+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:55:50.580 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.577+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:55:50.580 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.577+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:55:50.580 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.577+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:55:50.580 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.577+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:55:50.580 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.577+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:55:50.580 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.577+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:55:50.580 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.577+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:55:50.581 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.577+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:55:50.581 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.577+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:55:50.585 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.581+0000 7f8dbea41940 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:55:50.585 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.581+0000 7f8dbea41940 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-02-05T04:55:50.585 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.581+0000 7f8dbea41940 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-02-05T04:55:50.585 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.581+0000 7f8dbea41940 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-05T04:55:50.585 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.581+0000 7f8dbea41940 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-02-05T04:55:50.585 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.581+0000 7f8dbea41940 1 freelist init 2024-02-05T04:55:50.585 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.581+0000 7f8dbea41940 1 freelist _read_cfg 2024-02-05T04:55:50.586 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.581+0000 7f8dbea41940 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 15 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc2000, fragmentation 5.34059e-07 2024-02-05T04:55:50.586 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.581+0000 7f8dbea41940 1 bluefs umount 2024-02-05T04:55:50.586 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.581+0000 7f8dbea41940 1 bdev(0x55ff64b60700 td/osd-mapper/1/block) close 2024-02-05T04:55:50.851 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.849+0000 7f8dbea41940 1 bdev(0x55ff64b60700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-02-05T04:55:50.851 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.849+0000 7f8dbea41940 1 bdev(0x55ff64b60700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:50.851 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.849+0000 7f8dbea41940 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-02-05T04:55:50.851 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.849+0000 7f8dbea41940 1 bluefs mount 2024-02-05T04:55:50.858 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.857+0000 7f8dbea41940 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:55:50.868 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.865+0000 7f8dbea41940 1 bluefs mount shared_bdev_used = 9568256 2024-02-05T04:55:50.868 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.865+0000 7f8dbea41940 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:55:50.872 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.869+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:55:50.872 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.869+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:55:50.873 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.869+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:55:50.873 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.869+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:55:50.873 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.869+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:55:50.873 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.869+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:55:50.873 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.869+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:55:50.873 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.869+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:55:50.873 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.869+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:55:50.873 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.869+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:55:50.873 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.869+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:55:50.873 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:50.869+0000 7f8dbea41940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:55:51.125 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:51.121+0000 7f8dbea41940 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:55:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-05T04:55:51.127 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 0b 05 58 02 00 00 42 95 50 96 ca b3 4e 29 ab 89 |..X...B.P...N)..| 2024-02-05T04:55:51.128 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 af 00 2f 47 44 56 01 00 00 00 1b 00 00 00 00 00 |../GDV..........| 2024-02-05T04:55:51.128 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-05T04:55:51.128 INFO:tasks.workunit.client.0.smithi148.stdout:* 2024-02-05T04:55:51.128 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-05T04:55:51.128 INFO:tasks.workunit.client.0.smithi148.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-05T04:55:51.128 INFO:tasks.workunit.client.0.smithi148.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-05T04:55:51.128 INFO:tasks.workunit.client.0.smithi148.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-05T04:55:51.128 INFO:tasks.workunit.client.0.smithi148.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-05T04:55:51.128 INFO:tasks.workunit.client.0.smithi148.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-05T04:55:51.128 INFO:tasks.workunit.client.0.smithi148.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-05T04:55:51.128 INFO:tasks.workunit.client.0.smithi148.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-05T04:55:51.128 INFO:tasks.workunit.client.0.smithi148.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-05T04:55:51.129 INFO:tasks.workunit.client.0.smithi148.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-05T04:55:51.130 INFO:tasks.workunit.client.0.smithi148.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-05T04:55:51.130 INFO:tasks.workunit.client.0.smithi148.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-05T04:55:51.130 INFO:tasks.workunit.client.0.smithi148.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-05T04:55:51.130 INFO:tasks.workunit.client.0.smithi148.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-05T04:55:51.130 INFO:tasks.workunit.client.0.smithi148.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-05T04:55:51.130 INFO:tasks.workunit.client.0.smithi148.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-05T04:55:51.130 INFO:tasks.workunit.client.0.smithi148.stdout:00000220 72 65 1b 00 00 00 0a 00 00 00 a8 18 24 e4 d1 60 |re..........$..`| 2024-02-05T04:55:51.130 INFO:tasks.workunit.client.0.smithi148.stdout:00000230 4b de a9 38 51 95 81 18 15 62 00 00 00 00 00 00 |K..8Q....b......| 2024-02-05T04:55:51.130 INFO:tasks.workunit.client.0.smithi148.stdout:00000240 00 00 1b 00 00 00 14 6a c0 65 e1 f4 2c 28 01 00 |.......j.e..,(..| 2024-02-05T04:55:51.130 INFO:tasks.workunit.client.0.smithi148.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1b 00 00 00 |..............| 2024-02-05T04:55:51.130 INFO:tasks.workunit.client.0.smithi148.stdout:0000025e 2024-02-05T04:55:51.130 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:51.130 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-05T04:55:51.130 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:51.131 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:51.131 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:55:51.131 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:51.131 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:51.131 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-05T04:55:51.131 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:55:51.131 INFO:tasks.workunit.client.0.smithi148.stdout:00000047 2024-02-05T04:55:51.131 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:51.131 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:51.131 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-05T04:55:51.131 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:51.131 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:51.131 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-05T04:55:51.131 INFO:tasks.workunit.client.0.smithi148.stdout:0000003f 2024-02-05T04:55:51.132 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:51.132 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:51.132 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:55:51.132 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:51.132 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:51.132 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-05T04:55:51.132 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:55:51.132 INFO:tasks.workunit.client.0.smithi148.stdout:00000047 2024-02-05T04:55:51.132 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:51.132 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:51.132 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:51.132 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:51.132 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:51.132 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:51.133 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:51.133 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:51.133 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:51.133 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:51.133 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:51.133 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:51.133 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:51.133 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:51.133 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:51.133 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:51.133 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:51.133 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:51.133 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:51.133 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:51.134 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:51.135 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:51.135 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-05T04:55:51.135 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:51.135 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-05T04:55:51.135 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:51.135 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:51.125+0000 7f8dbea41940 1 bluefs umount 2024-02-05T04:55:51.135 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:51.125+0000 7f8dbea41940 1 bdev(0x55ff64b60700 td/osd-mapper/1/block) close 2024-02-05T04:55:51.378 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:51.377+0000 7f8dbea41940 1 freelist shutdown 2024-02-05T04:55:51.379 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:51.377+0000 7f8dbea41940 1 bdev(0x55ff64b60000 td/osd-mapper/1/block) close 2024-02-05T04:55:51.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-05T04:55:51.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-02-05T04:55:51.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-05T04:55:52.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:52.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:55:52.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-02-05T04:55:52.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-02-05T04:55:52.679 INFO:tasks.workunit.client.0.smithi148.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:52.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-02-05T04:55:52.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.ffiYremO5A_the_val 2024-02-05T04:55:52.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:55:52.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.ffiYremO5A_the_val' 2024-02-05T04:55:52.681 INFO:tasks.workunit.client.0.smithi148.stdout:Value dumped in: /tmp/tmp.ffiYremO5A_the_val 2024-02-05T04:55:52.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.ffiYremO5A_the_val 2024-02-05T04:55:53.219 INFO:tasks.workunit.client.0.smithi148.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-02-05T04:55:53.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:55:53.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.ffiYremO5A_the_val 2024-02-05T04:55:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-02-05T04:55:53.754 INFO:tasks.workunit.client.0.smithi148.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-02-05T04:55:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-02-05T04:55:53.754 INFO:tasks.workunit.client.0.smithi148.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-02-05T04:55:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-02-05T04:55:53.754 INFO:tasks.workunit.client.0.smithi148.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-02-05T04:55:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-02-05T04:55:53.754 INFO:tasks.workunit.client.0.smithi148.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-02-05T04:55:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:0000073 2024-02-05T04:55:53.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-02-05T04:55:53.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:54.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.ffiYremO5A_the_val 2024-02-05T04:55:55.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:55:55.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:55:55.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-02-05T04:55:55.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-02-05T04:55:55.901 INFO:tasks.workunit.client.0.smithi148.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-02-05T04:55:55.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-02-05T04:55:55.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-02-05T04:55:55.929 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.924+0000 7efc9ed69940 1 bdev(0x559dcb5ee000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-05T04:55:55.930 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.924+0000 7efc9ed69940 1 bdev(0x559dcb5ee000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:55.930 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.924+0000 7efc9ed69940 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-05T04:55:55.930 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.924+0000 7efc9ed69940 1 bdev(0x559dcb5ee700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-05T04:55:55.930 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.924+0000 7efc9ed69940 1 bdev(0x559dcb5ee700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:55.930 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.924+0000 7efc9ed69940 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-02-05T04:55:55.930 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.924+0000 7efc9ed69940 1 bluefs mount 2024-02-05T04:55:55.933 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.928+0000 7efc9ed69940 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:55:55.942 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.936+0000 7efc9ed69940 1 bluefs mount shared_bdev_used = 0 2024-02-05T04:55:55.942 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.940+0000 7efc9ed69940 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:55:55.946 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.940+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:55:55.946 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.940+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:55:55.946 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.940+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:55:55.946 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.940+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:55:55.946 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.940+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:55:55.946 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.940+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:55:55.946 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.940+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:55:55.946 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.940+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:55:55.946 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.940+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:55:55.946 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.940+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:55:55.946 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.940+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:55:55.946 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.940+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:55:55.951 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.948+0000 7efc9ed69940 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:55:55.951 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.948+0000 7efc9ed69940 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-02-05T04:55:55.952 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.948+0000 7efc9ed69940 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-02-05T04:55:55.952 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.948+0000 7efc9ed69940 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-05T04:55:55.952 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.948+0000 7efc9ed69940 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-02-05T04:55:55.952 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.948+0000 7efc9ed69940 1 freelist init 2024-02-05T04:55:55.952 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.948+0000 7efc9ed69940 1 freelist _read_cfg 2024-02-05T04:55:55.952 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.948+0000 7efc9ed69940 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc2000, fragmentation 4.95912e-07 2024-02-05T04:55:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.948+0000 7efc9ed69940 1 bluefs umount 2024-02-05T04:55:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:55.948+0000 7efc9ed69940 1 bdev(0x559dcb5ee700 td/osd-mapper/2/block) close 2024-02-05T04:55:56.211 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.208+0000 7efc9ed69940 1 bdev(0x559dcb5ee700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-05T04:55:56.211 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.208+0000 7efc9ed69940 1 bdev(0x559dcb5ee700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:55:56.211 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.208+0000 7efc9ed69940 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-02-05T04:55:56.211 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.208+0000 7efc9ed69940 1 bluefs mount 2024-02-05T04:55:56.216 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.212+0000 7efc9ed69940 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:55:56.226 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.220+0000 7efc9ed69940 1 bluefs mount shared_bdev_used = 9568256 2024-02-05T04:55:56.226 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.220+0000 7efc9ed69940 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:55:56.229 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.224+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:55:56.229 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.224+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:55:56.229 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.224+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:55:56.229 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.224+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:55:56.229 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.224+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:55:56.230 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.224+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:55:56.230 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.224+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:55:56.230 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.224+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:55:56.230 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.224+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:55:56.230 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.224+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:55:56.230 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.224+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:55:56.230 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.224+0000 7efc9ed69940 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:55:56.450 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.448+0000 7efc9ed69940 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:55:56.450 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-05T04:55:56.452 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 0b 05 58 02 00 00 42 95 50 96 ca b3 4e 29 ab 89 |..X...B.P...N)..| 2024-02-05T04:55:56.453 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 af 00 2f 47 44 56 02 00 00 00 1b 00 00 00 00 00 |../GDV..........| 2024-02-05T04:55:56.453 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-05T04:55:56.453 INFO:tasks.workunit.client.0.smithi148.stdout:* 2024-02-05T04:55:56.453 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-05T04:55:56.453 INFO:tasks.workunit.client.0.smithi148.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-05T04:55:56.453 INFO:tasks.workunit.client.0.smithi148.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-05T04:55:56.453 INFO:tasks.workunit.client.0.smithi148.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-05T04:55:56.453 INFO:tasks.workunit.client.0.smithi148.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-05T04:55:56.453 INFO:tasks.workunit.client.0.smithi148.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-05T04:55:56.453 INFO:tasks.workunit.client.0.smithi148.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-05T04:55:56.453 INFO:tasks.workunit.client.0.smithi148.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-05T04:55:56.453 INFO:tasks.workunit.client.0.smithi148.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-05T04:55:56.454 INFO:tasks.workunit.client.0.smithi148.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-05T04:55:56.454 INFO:tasks.workunit.client.0.smithi148.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-05T04:55:56.454 INFO:tasks.workunit.client.0.smithi148.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-05T04:55:56.454 INFO:tasks.workunit.client.0.smithi148.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-05T04:55:56.454 INFO:tasks.workunit.client.0.smithi148.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-05T04:55:56.454 INFO:tasks.workunit.client.0.smithi148.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-05T04:55:56.454 INFO:tasks.workunit.client.0.smithi148.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-05T04:55:56.454 INFO:tasks.workunit.client.0.smithi148.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-05T04:55:56.454 INFO:tasks.workunit.client.0.smithi148.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-05T04:55:56.454 INFO:tasks.workunit.client.0.smithi148.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-05T04:55:56.454 INFO:tasks.workunit.client.0.smithi148.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-05T04:55:56.454 INFO:tasks.workunit.client.0.smithi148.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-05T04:55:56.454 INFO:tasks.workunit.client.0.smithi148.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-05T04:55:56.455 INFO:tasks.workunit.client.0.smithi148.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-05T04:55:56.455 INFO:tasks.workunit.client.0.smithi148.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-05T04:55:56.455 INFO:tasks.workunit.client.0.smithi148.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-05T04:55:56.455 INFO:tasks.workunit.client.0.smithi148.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-05T04:55:56.455 INFO:tasks.workunit.client.0.smithi148.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-05T04:55:56.455 INFO:tasks.workunit.client.0.smithi148.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-05T04:55:56.455 INFO:tasks.workunit.client.0.smithi148.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-05T04:55:56.455 INFO:tasks.workunit.client.0.smithi148.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-05T04:55:56.455 INFO:tasks.workunit.client.0.smithi148.stdout:00000220 72 65 1b 00 00 00 0f 00 00 00 0d 8c 91 4c 2a 20 |re...........L* | 2024-02-05T04:55:56.455 INFO:tasks.workunit.client.0.smithi148.stdout:00000230 4f 16 84 4b 9c 10 f9 45 c6 be 00 00 00 00 00 00 |O..K...E........| 2024-02-05T04:55:56.455 INFO:tasks.workunit.client.0.smithi148.stdout:00000240 00 00 1b 00 00 00 35 6a c0 65 70 7b 42 39 01 00 |......5j.ep{B9..| 2024-02-05T04:55:56.455 INFO:tasks.workunit.client.0.smithi148.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1b 00 00 00 |..............| 2024-02-05T04:55:56.455 INFO:tasks.workunit.client.0.smithi148.stdout:0000025e 2024-02-05T04:55:56.455 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:56.456 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-05T04:55:56.456 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:56.456 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:56.456 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:55:56.456 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:56.456 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:56.456 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-05T04:55:56.456 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:55:56.456 INFO:tasks.workunit.client.0.smithi148.stdout:00000047 2024-02-05T04:55:56.456 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:56.456 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:56.456 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-05T04:55:56.457 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:56.457 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:56.457 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-05T04:55:56.457 INFO:tasks.workunit.client.0.smithi148.stdout:0000003f 2024-02-05T04:55:56.457 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:56.457 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:56.457 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:55:56.457 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-05T04:55:56.457 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:55:56.457 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-05T04:55:56.457 INFO:tasks.workunit.client.0.smithi148.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:55:56.457 INFO:tasks.workunit.client.0.smithi148.stdout:00000047 2024-02-05T04:55:56.457 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:56.457 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:56.458 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:56.458 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:56.458 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:56.458 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:56.458 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:56.458 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:56.458 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:55:56.458 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:56.458 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:56.458 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:56.458 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:56.458 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:56.458 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:56.459 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:56.459 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:56.459 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:56.459 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:56.459 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:56.459 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:56.459 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:56.459 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:56.459 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:56.460 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:56.460 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:56.460 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:56.460 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:56.460 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:56.460 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:55:56.460 INFO:tasks.workunit.client.0.smithi148.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:55:56.460 INFO:tasks.workunit.client.0.smithi148.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:55:56.460 INFO:tasks.workunit.client.0.smithi148.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:55:56.460 INFO:tasks.workunit.client.0.smithi148.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:55:56.460 INFO:tasks.workunit.client.0.smithi148.stdout:0000003b 2024-02-05T04:55:56.460 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:56.460 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-05T04:55:56.461 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:56.461 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-05T04:55:56.461 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:55:56.461 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.448+0000 7efc9ed69940 1 bluefs umount 2024-02-05T04:55:56.461 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.448+0000 7efc9ed69940 1 bdev(0x559dcb5ee700 td/osd-mapper/2/block) close 2024-02-05T04:55:56.706 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.704+0000 7efc9ed69940 1 freelist shutdown 2024-02-05T04:55:56.707 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:55:56.704+0000 7efc9ed69940 1 bdev(0x559dcb5ee000 td/osd-mapper/2/block) close 2024-02-05T04:55:56.964 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-02-05T04:55:56.964 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-02-05T04:55:56.964 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-05T04:55:58.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:58.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:55:58.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-02-05T04:55:58.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-02-05T04:55:58.031 INFO:tasks.workunit.client.0.smithi148.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:55:58.032 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-02-05T04:55:58.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.IyU3JG37tC_the_val 2024-02-05T04:55:58.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:55:58.033 INFO:tasks.workunit.client.0.smithi148.stdout:Value dumped in: /tmp/tmp.IyU3JG37tC_the_val 2024-02-05T04:55:58.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.IyU3JG37tC_the_val' 2024-02-05T04:55:58.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.IyU3JG37tC_the_val 2024-02-05T04:55:58.561 INFO:tasks.workunit.client.0.smithi148.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-02-05T04:55:59.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:55:59.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.IyU3JG37tC_the_val 2024-02-05T04:55:59.100 INFO:tasks.workunit.client.0.smithi148.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-02-05T04:55:59.100 INFO:tasks.workunit.client.0.smithi148.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-02-05T04:55:59.100 INFO:tasks.workunit.client.0.smithi148.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-02-05T04:55:59.100 INFO:tasks.workunit.client.0.smithi148.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-02-05T04:55:59.100 INFO:tasks.workunit.client.0.smithi148.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-02-05T04:55:59.100 INFO:tasks.workunit.client.0.smithi148.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-02-05T04:55:59.100 INFO:tasks.workunit.client.0.smithi148.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-02-05T04:55:59.101 INFO:tasks.workunit.client.0.smithi148.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-02-05T04:55:59.101 INFO:tasks.workunit.client.0.smithi148.stdout:0000073 2024-02-05T04:55:59.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-02-05T04:55:59.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:56:00.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.IyU3JG37tC_the_val 2024-02-05T04:56:01.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:56:01.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 ' 2024-02-05T04:56:01.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 2024-02-05T04:56:01.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2024-02-05T04:56:01.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:56:01.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/' 2024-02-05T04:56:01.487 INFO:tasks.workunit.client.0.smithi148.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/ 2024-02-05T04:56:01.488 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-02-05T04:56:01.501 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-02-05T04:56:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:56:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=42955096-cab3-4e29-ab89-af002f474456 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2024-02-05T04:56:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-02-05T04:56:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-02-05T04:56:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:56:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:56:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:56:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2024-02-05T04:56:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=42955096-cab3-4e29-ab89-af002f474456 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2024-02-05T04:56:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:56:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:56:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:56:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-02-05T04:56:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-02-05T04:56:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:56:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:56:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20103 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20103/$cluster-$name.asok' 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20103/$cluster-$name.asok' 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:56:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:56:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:56:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T04:56:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2024-02-05T04:56:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:56:01.507 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T04:56:01.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=42955096-cab3-4e29-ab89-af002f474456 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20103/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T04:56:01.508 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2024-02-05T04:56:01.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:56:01.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:56:01.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:56:01.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:56:01.539 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:01.532+0000 7f87e6899640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:01.541 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:01.536+0000 7f87e6899640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:01.543 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:01.536+0000 7f87e6899640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:01.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:56:01.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:56:01.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:56:01.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:56:01.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:56:01.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:01.825 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T04:56:01.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:56:01.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:01.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:02.077 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:02.072+0000 7f87e6899640 -1 Falling back to public interface 2024-02-05T04:56:02.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:03.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:03.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:03.143 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T04:56:03.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:56:03.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:03.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:03.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:04.459 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T04:56:04.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:04.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:04.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:56:04.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:04.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:04.493 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:04.488+0000 7f87e6899640 -1 osd.0 27 log_to_monitors true 2024-02-05T04:56:04.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:05.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:05.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:05.798 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T04:56:05.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:56:05.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:05.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:06.119 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 30 up_thru 30 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6800/1413254793,v1:127.0.0.1:6801/1413254793] [v2:127.0.0.1:6802/1413254793,v1:127.0.0.1:6803/1413254793] exists,up 44def9ba-32d8-4313-b826-77c7fca29a17 2024-02-05T04:56:06.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:56:06.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:56:06.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:56:06.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:56:06.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=42955096-cab3-4e29-ab89-af002f474456 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2024-02-05T04:56:06.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-02-05T04:56:06.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-02-05T04:56:06.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:56:06.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T04:56:06.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:56:06.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2024-02-05T04:56:06.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=42955096-cab3-4e29-ab89-af002f474456 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2024-02-05T04:56:06.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:56:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:56:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:56:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-02-05T04:56:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-02-05T04:56:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:56:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:56:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-02-05T04:56:06.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:56:06.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:56:06.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:56:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:56:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20103 2024-02-05T04:56:06.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20103/$cluster-$name.asok' 2024-02-05T04:56:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20103/$cluster-$name.asok' 2024-02-05T04:56:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:56:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-02-05T04:56:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-02-05T04:56:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:56:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:56:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:56:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:56:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:56:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T04:56:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2024-02-05T04:56:06.125 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T04:56:06.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T04:56:06.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=42955096-cab3-4e29-ab89-af002f474456 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20103/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T04:56:06.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2024-02-05T04:56:06.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T04:56:06.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:56:06.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:56:06.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:56:06.164 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:06.160+0000 7fe3a1956640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:06.177 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:06.172+0000 7fe3a1956640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:06.178 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:06.172+0000 7fe3a1956640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:06.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T04:56:06.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:56:06.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:56:06.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:56:06.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:56:06.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:06.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:56:06.441 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T04:56:06.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:06.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:56:06.705 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:06.700+0000 7fe3a1956640 -1 Falling back to public interface 2024-02-05T04:56:06.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:07.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:07.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:07.744 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T04:56:07.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:56:07.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:07.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:56:08.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:09.130 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T04:56:09.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:09.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:09.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:56:09.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:09.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:56:09.189 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:09.184+0000 7fe3a1956640 -1 osd.1 27 log_to_monitors true 2024-02-05T04:56:09.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:10.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:10.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:10.507 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T04:56:10.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:56:10.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:10.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:56:10.839 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 33 up_thru 33 down_at 28 last_clean_interval [10,27) [v2:127.0.0.1:6810/413049744,v1:127.0.0.1:6811/413049744] [v2:127.0.0.1:6812/413049744,v1:127.0.0.1:6813/413049744] exists,up a81824e4-d160-4bde-a938-519581181562 2024-02-05T04:56:10.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:56:10.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:56:10.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:56:10.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:56:10.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=42955096-cab3-4e29-ab89-af002f474456 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2024-02-05T04:56:10.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-02-05T04:56:10.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-02-05T04:56:10.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:56:10.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T04:56:10.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:56:10.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2024-02-05T04:56:10.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=42955096-cab3-4e29-ab89-af002f474456 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2024-02-05T04:56:10.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:56:10.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:56:10.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:56:10.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-02-05T04:56:10.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-02-05T04:56:10.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:56:10.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:56:10.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-02-05T04:56:10.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:56:10.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:56:10.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:56:10.843 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:56:10.844 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:10.844 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20103 2024-02-05T04:56:10.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20103/$cluster-$name.asok' 2024-02-05T04:56:10.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20103/$cluster-$name.asok' 2024-02-05T04:56:10.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:56:10.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-02-05T04:56:10.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-02-05T04:56:10.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:56:10.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:56:10.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:56:10.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:56:10.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:56:10.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T04:56:10.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2024-02-05T04:56:10.845 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T04:56:10.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T04:56:10.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=42955096-cab3-4e29-ab89-af002f474456 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20103/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T04:56:10.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2024-02-05T04:56:10.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T04:56:10.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:56:10.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:56:10.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:56:10.884 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:10.880+0000 7f3cae868640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:10.891 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:10.888+0000 7f3cae868640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:10.892 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:10.888+0000 7f3cae868640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:11.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T04:56:11.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:56:11.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T04:56:11.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:56:11.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:56:11.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:11.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:56:11.162 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T04:56:11.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:11.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:56:11.429 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:11.424+0000 7f3cae868640 -1 Falling back to public interface 2024-02-05T04:56:11.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:12.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:12.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:12.478 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T04:56:12.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:56:12.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:12.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:56:12.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:13.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:13.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:13.813 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T04:56:13.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:56:13.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:13.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:56:13.947 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:13.944+0000 7f3cae868640 -1 osd.2 27 log_to_monitors true 2024-02-05T04:56:14.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:15.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:15.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:15.296 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T04:56:15.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:56:15.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:15.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:56:15.756 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 36 up_thru 36 down_at 28 last_clean_interval [15,27) [v2:127.0.0.1:6818/808503108,v1:127.0.0.1:6819/808503108] [v2:127.0.0.1:6820/808503108,v1:127.0.0.1:6821/808503108] exists,up 0d8c914c-2a20-4f16-844b-9c10f945c6be 2024-02-05T04:56:15.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:56:15.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:56:15.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:56:15.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-02-05T04:56:16.759 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-02-05T04:56:16.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-02-05T04:56:16.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:56:16.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-02-05T04:56:16.902 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T04:56:16.903 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "19.0.0-658-gd4ed20cf", 2024-02-05T04:56:16.903 INFO:tasks.workunit.client.0.smithi148.stdout: "release": "squid", 2024-02-05T04:56:16.903 INFO:tasks.workunit.client.0.smithi148.stdout: "release_type": "dev" 2024-02-05T04:56:16.903 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T04:56:16.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:56:16.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-02-05T04:56:17.045 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T04:56:17.045 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "19.0.0-658-gd4ed20cf", 2024-02-05T04:56:17.045 INFO:tasks.workunit.client.0.smithi148.stdout: "release": "squid", 2024-02-05T04:56:17.045 INFO:tasks.workunit.client.0.smithi148.stdout: "release_type": "dev" 2024-02-05T04:56:17.045 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T04:56:17.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:56:17.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-02-05T04:56:17.184 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T04:56:17.185 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "19.0.0-658-gd4ed20cf", 2024-02-05T04:56:17.185 INFO:tasks.workunit.client.0.smithi148.stdout: "release": "squid", 2024-02-05T04:56:17.185 INFO:tasks.workunit.client.0.smithi148.stdout: "release_type": "dev" 2024-02-05T04:56:17.185 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T04:56:17.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-02-05T04:56:17.225 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T04:56:17.226 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "objxxx", 2024-02-05T04:56:17.226 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T04:56:17.226 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:56:17.226 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 2, 2024-02-05T04:56:17.226 INFO:tasks.workunit.client.0.smithi148.stdout: "snapshots": [ 2024-02-05T04:56:17.226 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:56:17.226 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 1, 2024-02-05T04:56:17.226 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "snap01" 2024-02-05T04:56:17.226 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T04:56:17.226 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:56:17.226 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 2, 2024-02-05T04:56:17.226 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "snap02" 2024-02-05T04:56:17.227 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T04:56:17.227 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T04:56:17.227 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 5, 2024-02-05T04:56:17.227 INFO:tasks.workunit.client.0.smithi148.stdout: "overlaps": [] 2024-02-05T04:56:17.227 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T04:56:17.227 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:56:17.227 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 3, 2024-02-05T04:56:17.227 INFO:tasks.workunit.client.0.smithi148.stdout: "snapshots": [ 2024-02-05T04:56:17.227 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:56:17.227 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 3, 2024-02-05T04:56:17.227 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "snap13" 2024-02-05T04:56:17.227 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T04:56:17.227 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T04:56:17.227 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 3, 2024-02-05T04:56:17.228 INFO:tasks.workunit.client.0.smithi148.stdout: "overlaps": [] 2024-02-05T04:56:17.228 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T04:56:17.228 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:56:17.228 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 5, 2024-02-05T04:56:17.228 INFO:tasks.workunit.client.0.smithi148.stdout: "snapshots": [ 2024-02-05T04:56:17.228 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:56:17.228 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 4, 2024-02-05T04:56:17.228 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "snap24" 2024-02-05T04:56:17.228 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T04:56:17.228 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:56:17.228 INFO:tasks.workunit.client.0.smithi148.stdout: "id": 5, 2024-02-05T04:56:17.229 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "snap25" 2024-02-05T04:56:17.229 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T04:56:17.229 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T04:56:17.229 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 5, 2024-02-05T04:56:17.229 INFO:tasks.workunit.client.0.smithi148.stdout: "overlaps": [] 2024-02-05T04:56:17.229 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T04:56:17.229 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T04:56:17.229 INFO:tasks.workunit.client.0.smithi148.stdout: "id": "head", 2024-02-05T04:56:17.229 INFO:tasks.workunit.client.0.smithi148.stdout: "snapshots": [], 2024-02-05T04:56:17.229 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 6 2024-02-05T04:56:17.229 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T04:56:17.229 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T04:56:17.230 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T04:56:17.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-02-05T04:56:17.593 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is unset 2024-02-05T04:56:17.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-02-05T04:56:18.129 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is unset 2024-02-05T04:56:18.151 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-02-05T04:56:18.151 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-02-05T04:56:18.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-02-05T04:56:18.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-05T04:56:18.758 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:56:18.759 INFO:tasks.workunit.client.0.smithi148.stdout:1.3 4 0 4 0 0 19 0 0 7 0 7 active+undersized+degraded 2024-02-05T04:56:10.245699+0000 25'7 34:45 [1,0] 1 [1,0] 1 25'7 2024-02-05T04:55:29.953575+0000 25'7 2024-02-05T04:55:29.953575+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:55:29.953575+0000 4 0 2024-02-05T04:56:18.759 INFO:tasks.workunit.client.0.smithi148.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 peering 2024-02-05T04:56:14.643266+0000 0'0 34:25 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 no scrub is scheduled 0 0 2024-02-05T04:56:18.759 INFO:tasks.workunit.client.0.smithi148.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 peering 2024-02-05T04:56:09.652298+0000 0'0 33:11 [0,1] 0 [0,1] 0 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 no scrub is scheduled 0 0 2024-02-05T04:56:18.759 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-02-05T04:56:10.170466+0000 0'0 34:19 [1,0] 1 [1,0] 1 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:55:18.910100+0000 0 0 2024-02-05T04:56:18.759 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:56:18.759 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:56:18.759 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T04:56:18.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-02-05T04:56:20.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-02-05T04:56:20.914 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T04:56:20.914 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": true, 2024-02-05T04:56:20.914 INFO:tasks.workunit.client.0.smithi148.stdout: "must": true, 2024-02-05T04:56:20.914 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "0.000000" 2024-02-05T04:56:20.914 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T04:56:20.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-02-05T04:56:25.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-05T04:56:26.219 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:56:26.219 INFO:tasks.workunit.client.0.smithi148.stdout:1.3 4 0 0 0 0 19 0 0 7 0 7 active+clean 2024-02-05T04:56:21.355420+0000 25'7 40:68 [1,2,0] 1 [1,2,0] 1 25'7 2024-02-05T04:56:21.355128+0000 25'7 2024-02-05T04:56:21.355128+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:56:21.355128+0000 4 0 2024-02-05T04:56:26.219 INFO:tasks.workunit.client.0.smithi148.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:56:16.276810+0000 0'0 39:37 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:55:18.910100+0000 0 0 2024-02-05T04:56:26.219 INFO:tasks.workunit.client.0.smithi148.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:56:15.196535+0000 0'0 40:26 [2,0,1] 2 [2,0,1] 2 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:55:18.910100+0000 0 0 2024-02-05T04:56:26.219 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:56:15.230049+0000 0'0 40:32 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:55:18.910100+0000 0 0 2024-02-05T04:56:26.220 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:56:26.220 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:56:26.220 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T04:56:26.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:56:26.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-02-05T04:56:26.239 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T04:56:21.352+0000 7fe3840dd640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-02-05T04:56:26.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-02-05T04:56:26.241 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-02-05T04:56:26.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-02-05T04:56:26.244 INFO:tasks.workunit.client.0.smithi148.stdout:prev count: 1 2024-02-05T04:56:26.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-02-05T04:56:26.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-02-05T04:56:26.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-02-05T04:56:26.382 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T04:56:26.382 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": true, 2024-02-05T04:56:26.382 INFO:tasks.workunit.client.0.smithi148.stdout: "must": true, 2024-02-05T04:56:26.382 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "0.000000" 2024-02-05T04:56:26.382 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T04:56:26.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-02-05T04:56:31.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-05T04:56:31.693 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:56:31.693 INFO:tasks.workunit.client.0.smithi148.stdout:1.3 4 0 0 0 0 19 0 0 7 0 7 active+clean 2024-02-05T04:56:27.243823+0000 25'7 40:77 [1,2,0] 1 [1,2,0] 1 25'7 2024-02-05T04:56:27.243521+0000 25'7 2024-02-05T04:56:27.243521+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:56:27.243521+0000 4 0 2024-02-05T04:56:31.693 INFO:tasks.workunit.client.0.smithi148.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:56:16.276810+0000 0'0 39:37 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:55:18.910100+0000 0 0 2024-02-05T04:56:31.693 INFO:tasks.workunit.client.0.smithi148.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:56:15.196535+0000 0'0 40:26 [2,0,1] 2 [2,0,1] 2 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:55:18.910100+0000 0 0 2024-02-05T04:56:31.693 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:56:15.230049+0000 0'0 40:32 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-05T04:55:18.910100+0000 0'0 2024-02-05T04:55:18.910100+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:55:18.910100+0000 0 0 2024-02-05T04:56:31.693 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:56:31.693 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:56:31.694 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T04:56:31.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:56:31.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-02-05T04:56:31.712 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T04:56:21.352+0000 7fe3840dd640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-02-05T04:56:31.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-02-05T04:56:31.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-02-05T04:56:31.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-02-05T04:56:31.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:56:31.715 INFO:tasks.workunit.client.0.smithi148.stdout:current count: 1 2024-02-05T04:56:31.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-02-05T04:56:31.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-02-05T04:56:31.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-02-05T04:56:31.716 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:56:31.717 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:56:31.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:56:31.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:56:31.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:56:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:56:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-02-05T04:56:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:56:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-05T04:56:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-02-05T04:56:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-05T04:56:33.348 INFO:tasks.workunit.client.0.smithi148.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:56:33.348 INFO:tasks.workunit.client.0.smithi148.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:56:33.348 INFO:tasks.workunit.client.0.smithi148.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-02-05T04:56:33.348 INFO:tasks.workunit.client.0.smithi148.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:56:33.348 INFO:tasks.workunit.client.0.smithi148.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:56:33.348 INFO:tasks.workunit.client.0.smithi148.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:56:33.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-05T04:56:33.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-02-05T04:56:33.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-05T04:56:33.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-02-05T04:56:34.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-02-05T04:56:34.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-02-05T04:56:34.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-02-05T04:56:34.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-02-05T04:56:34.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-02-05T04:56:34.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:56:34.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-02-05T04:56:34.415 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:56:34.415 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:56:34.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:56:34.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:56:34.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:56:34.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:56:34.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:56:34.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:56:34.526 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:56:34.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:56:34.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:56:34.527 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:56:34.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:56:34.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:56:34.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:56:34.529 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:56:34.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:56:34.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:56:34.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-02-05T04:56:34.549 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:56:34.549 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:34.549 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20103 2024-02-05T04:56:34.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20103 2024-02-05T04:56:34.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:56:34.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:56:34.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-05T04:56:34.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-mapper 0 2024-02-05T04:56:34.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-02-05T04:56:34.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-05T04:56:34.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-02-05T04:56:34.551 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:56:34.551 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:56:34.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:56:34.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:56:34.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:56:34.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:56:34.553 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:56:34.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:56:34.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:56:34.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:56:34.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:56:34.556 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:56:34.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:56:34.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:56:34.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:56:34.557 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:56:34.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:56:34.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-05T04:56:34.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-02-05T04:56:34.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:56:34.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:34.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20103 2024-02-05T04:56:34.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20103 2024-02-05T04:56:34.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:56:34.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:56:34.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-05T04:56:34.562 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-05T04:56:34.563 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-05T04:56:34.618 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-02-05T04:56:34.618 DEBUG:teuthology.orchestra.run.smithi148:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-02-05T04:56:34.676 INFO:tasks.workunit.client.0.smithi148.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-05T04:56:34.681 INFO:tasks.workunit.client.0.smithi148.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-05T04:56:34.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-05T04:56:34.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-05T04:56:34.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-05T04:56:34.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-05T04:56:34.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-05T04:56:34.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-05T04:56:34.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-05T04:56:34.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-05T04:56:34.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-recovery-scrub 2024-02-05T04:56:34.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-02-05T04:56:34.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-02-05T04:56:34.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-02-05T04:56:34.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-02-05T04:56:34.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-02-05T04:56:34.682 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-02-05T04:56:34.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none ' 2024-02-05T04:56:34.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-02-05T04:56:34.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-02-05T04:56:34.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-02-05T04:56:34.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-02-05T04:56:34.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-05T04:56:34.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-02-05T04:56:34.686 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_recovery_scrub_2' 2024-02-05T04:56:34.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-02-05T04:56:34.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-02-05T04:56:34.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-02-05T04:56:34.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-02-05T04:56:34.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.36362 2024-02-05T04:56:34.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-02-05T04:56:34.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-02-05T04:56:34.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-02-05T04:56:34.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-02-05T04:56:34.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-02-05T04:56:34.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-02-05T04:56:34.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-02-05T04:56:34.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-02-05T04:56:34.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:56:34.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-02-05T04:56:34.687 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:56:34.687 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:56:34.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:56:34.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:56:34.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:56:34.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:56:34.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:56:34.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:56:34.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:56:34.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:56:34.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:56:34.691 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:56:34.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:56:34.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:56:34.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:56:34.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:56:34.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:56:34.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:56:34.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-02-05T04:56:34.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:56:34.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:34.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T04:56:34.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36362 2024-02-05T04:56:34.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:56:34.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:56:34.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-02-05T04:56:34.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T04:56:34.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:34.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T04:56:34.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36362 2024-02-05T04:56:34.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T04:56:34.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-05T04:56:34.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-05T04:56:34.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T04:56:34.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-02-05T04:56:34.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T04:56:34.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-02-05T04:56:34.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T04:56:34.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T04:56:34.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T04:56:34.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-02-05T04:56:34.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T04:56:35.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T04:56:35.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:56:35.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:56:35.078 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:56:35.078 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:35.078 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T04:56:35.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T04:56:35.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T04:56:35.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T04:56:35.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T04:56:35.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:56:35.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:56:35.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T04:56:35.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T04:56:35.299 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:56:35.300 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:56:35.300 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:56:35.300 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:56:35.300 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:35.300 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T04:56:35.300 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36362/ceph-mon.a.asok 2024-02-05T04:56:35.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:56:35.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36362/ceph-mon.a.asok config get fsid 2024-02-05T04:56:35.419 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T04:56:35.419 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:56:35.420 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:56:35.420 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T04:56:35.420 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T04:56:35.420 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:56:35.420 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:56:35.420 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:56:35.421 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:56:35.421 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:35.421 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T04:56:35.421 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36362/ceph-mon.a.asok 2024-02-05T04:56:35.421 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:56:35.421 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36362/ceph-mon.a.asok config get mon_host 2024-02-05T04:56:35.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-02-05T04:56:35.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-02-05T04:56:35.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T04:56:35.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T04:56:35.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T04:56:35.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-02-05T04:56:35.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T04:56:36.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T04:56:36.417 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:56:36.417 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:56:36.417 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:56:36.418 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:36.418 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T04:56:36.418 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T04:56:36.418 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:56:36.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:56:36.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-02-05T04:56:36.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-02-05T04:56:36.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-02-05T04:56:36.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-02-05T04:56:36.455 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-02-05T04:56:36.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-02-05T04:56:36.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:56:36.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-02-05T04:56:36.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:56:36.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:56:36.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T04:56:36.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:56:36.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-02-05T04:56:36.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:56:36.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:56:36.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:56:36.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:56:36.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-02-05T04:56:36.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-02-05T04:56:36.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:56:36.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:56:36.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:56:36.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:56:36.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:56:36.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:56:36.461 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:56:36.461 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:36.461 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T04:56:36.461 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T04:56:36.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T04:56:36.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:56:36.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:56:36.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:56:36.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:56:36.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:56:36.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:56:36.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:56:36.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:56:36.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:56:36.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:56:36.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-02-05T04:56:36.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-02-05T04:56:36.462 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:56:36.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e216229d-abd7-4424-8e60-21657dbf54f6 2024-02-05T04:56:36.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e216229d-abd7-4424-8e60-21657dbf54f6' 2024-02-05T04:56:36.463 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 e216229d-abd7-4424-8e60-21657dbf54f6 2024-02-05T04:56:36.464 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:56:36.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCEasBlDb9vHBAAUnisEmtFEZxTWeMyDHwC1g== 2024-02-05T04:56:36.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCEasBlDb9vHBAAUnisEmtFEZxTWeMyDHwC1g=="}' 2024-02-05T04:56:36.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e216229d-abd7-4424-8e60-21657dbf54f6 -i td/osd-recovery-scrub/0/new.json 2024-02-05T04:56:36.717 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T04:56:36.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-02-05T04:56:36.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCEasBlDb9vHBAAUnisEmtFEZxTWeMyDHwC1g== --osd-uuid e216229d-abd7-4424-8e60-21657dbf54f6 2024-02-05T04:56:36.771 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:36.768+0000 7ff4b77a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:36.774 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:36.768+0000 7ff4b77a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:36.775 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:36.772+0000 7ff4b77a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:36.775 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:36.772+0000 7ff4b77a0640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-05T04:56:36.775 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:36.772+0000 7ff4b77a0640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-05T04:56:36.776 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:36.772+0000 7ff4b77a0640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-05T04:56:36.852 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:36.848+0000 7ff4b77a0640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-02-05T04:56:42.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-02-05T04:56:42.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:56:42.765 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T04:56:42.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T04:56:42.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:56:43.134 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T04:56:43.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T04:56:43.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:56:43.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:56:43.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:56:43.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-02-05T04:56:43.166 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:43.159+0000 7f3f542e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:43.170 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:43.163+0000 7f3f542e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:43.171 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:43.167+0000 7f3f542e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:43.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T04:56:43.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:56:43.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:56:43.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:56:43.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:56:43.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:43.460 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T04:56:43.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:56:43.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:43.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:43.705 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:43.699+0000 7f3f542e5640 -1 Falling back to public interface 2024-02-05T04:56:43.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:44.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:44.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:44.788 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T04:56:44.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:56:44.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:44.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:45.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:46.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:46.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:46.092 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T04:56:46.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:56:46.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:46.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:46.195 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:46.191+0000 7f3f542e5640 -1 osd.0 0 log_to_monitors true 2024-02-05T04:56:46.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:47.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:47.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:47.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:56:47.473 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T04:56:47.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:47.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:47.787 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1342284735,v1:127.0.0.1:6803/1342284735] [v2:127.0.0.1:6804/1342284735,v1:127.0.0.1:6805/1342284735] exists,up e216229d-abd7-4424-8e60-21657dbf54f6 2024-02-05T04:56:47.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:56:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:56:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:56:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:56:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-02-05T04:56:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:56:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:56:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T04:56:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:56:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-02-05T04:56:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:56:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:56:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:56:47.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:56:47.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-02-05T04:56:47.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-02-05T04:56:47.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:56:47.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:56:47.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:56:47.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:56:47.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:56:47.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:56:47.792 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:56:47.792 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:47.792 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T04:56:47.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T04:56:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T04:56:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:56:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:56:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:56:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:56:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:56:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:56:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:56:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:56:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:56:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:56:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-02-05T04:56:47.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-02-05T04:56:47.794 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:56:47.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8cf6cdf6-1d64-4682-a76e-b76e3a9874a7 2024-02-05T04:56:47.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8cf6cdf6-1d64-4682-a76e-b76e3a9874a7' 2024-02-05T04:56:47.795 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 8cf6cdf6-1d64-4682-a76e-b76e3a9874a7 2024-02-05T04:56:47.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:56:47.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCPasBlE5Y6MBAAymagYo2vjqAsmAUfT/fKcQ== 2024-02-05T04:56:47.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCPasBlE5Y6MBAAymagYo2vjqAsmAUfT/fKcQ=="}' 2024-02-05T04:56:47.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8cf6cdf6-1d64-4682-a76e-b76e3a9874a7 -i td/osd-recovery-scrub/1/new.json 2024-02-05T04:56:48.156 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T04:56:48.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-02-05T04:56:48.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCPasBlE5Y6MBAAymagYo2vjqAsmAUfT/fKcQ== --osd-uuid 8cf6cdf6-1d64-4682-a76e-b76e3a9874a7 2024-02-05T04:56:48.208 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:48.203+0000 7faf07de5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:48.241 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:48.235+0000 7faf07de5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:48.243 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:48.239+0000 7faf07de5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:48.243 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:48.239+0000 7faf07de5640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-05T04:56:48.243 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:48.239+0000 7faf07de5640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-05T04:56:48.244 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:48.239+0000 7faf07de5640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-05T04:56:48.316 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:48.311+0000 7faf07de5640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-02-05T04:56:54.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-02-05T04:56:54.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:56:54.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T04:56:54.096 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T04:56:54.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:56:54.474 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T04:56:54.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T04:56:54.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:56:54.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-02-05T04:56:54.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:56:54.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:56:54.506 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:54.499+0000 7fc82c0ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:54.508 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:54.503+0000 7fc82c0ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:54.509 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:54.503+0000 7fc82c0ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:54.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T04:56:54.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:56:54.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:56:54.781 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T04:56:54.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:56:54.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:56:54.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:54.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:56:54.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:54.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:56:55.052 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:55.047+0000 7fc82c0ac640 -1 Falling back to public interface 2024-02-05T04:56:55.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:56.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:56.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:56.099 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T04:56:56.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:56:56.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:56.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:56:56.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:57.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:57.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:57.404 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T04:56:57.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:56:57.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:57.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:56:57.499 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:57.495+0000 7fc82c0ac640 -1 osd.1 0 log_to_monitors true 2024-02-05T04:56:57.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:58.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:58.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:58.759 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T04:56:58.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:56:58.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:58.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:56:59.075 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1408608852,v1:127.0.0.1:6811/1408608852] [v2:127.0.0.1:6812/1408608852,v1:127.0.0.1:6813/1408608852] exists,up 8cf6cdf6-1d64-4682-a76e-b76e3a9874a7 2024-02-05T04:56:59.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:56:59.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:56:59.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:56:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:56:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-02-05T04:56:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:56:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:56:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T04:56:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:56:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-02-05T04:56:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:56:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:56:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:56:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:56:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-02-05T04:56:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-02-05T04:56:59.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:56:59.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:56:59.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:56:59.079 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:56:59.079 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:56:59.079 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:56:59.079 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:56:59.079 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:59.079 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T04:56:59.081 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T04:56:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T04:56:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:56:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:56:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:56:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:56:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:56:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:56:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:56:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:56:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:56:59.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:56:59.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-02-05T04:56:59.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-02-05T04:56:59.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:56:59.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=736811a5-b2aa-4a25-9c44-e570a1b64129 2024-02-05T04:56:59.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 736811a5-b2aa-4a25-9c44-e570a1b64129' 2024-02-05T04:56:59.082 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 736811a5-b2aa-4a25-9c44-e570a1b64129 2024-02-05T04:56:59.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:56:59.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCbasBlpygMBhAA2xJn6hdPt0tQQSy95F1Nxw== 2024-02-05T04:56:59.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCbasBlpygMBhAA2xJn6hdPt0tQQSy95F1Nxw=="}' 2024-02-05T04:56:59.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 736811a5-b2aa-4a25-9c44-e570a1b64129 -i td/osd-recovery-scrub/2/new.json 2024-02-05T04:56:59.502 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T04:56:59.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-02-05T04:56:59.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCbasBlpygMBhAA2xJn6hdPt0tQQSy95F1Nxw== --osd-uuid 736811a5-b2aa-4a25-9c44-e570a1b64129 2024-02-05T04:56:59.555 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:59.551+0000 7f7dc4102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:59.582 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:59.575+0000 7f7dc4102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:59.583 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:59.579+0000 7f7dc4102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:59.584 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:59.579+0000 7f7dc4102640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-05T04:56:59.584 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:59.579+0000 7f7dc4102640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-05T04:56:59.584 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:59.579+0000 7f7dc4102640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-05T04:56:59.636 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:56:59.631+0000 7f7dc4102640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-02-05T04:57:05.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-02-05T04:57:05.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:57:05.406 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T04:57:05.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T04:57:05.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:57:05.795 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T04:57:05.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T04:57:05.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:57:05.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-02-05T04:57:05.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:57:05.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:57:05.829 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:05.823+0000 7fd01739d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:05.852 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:05.847+0000 7fd01739d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:05.853 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:05.847+0000 7fd01739d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:06.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T04:57:06.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:57:06.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T04:57:06.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:57:06.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:57:06.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:06.100 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T04:57:06.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:57:06.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:06.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:57:06.397 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:06.391+0000 7fd01739d640 -1 Falling back to public interface 2024-02-05T04:57:06.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:07.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:07.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:07.419 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T04:57:07.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:57:07.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:07.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:57:07.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:08.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:08.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:08.734 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T04:57:08.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:57:08.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:08.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:57:08.880 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:08.875+0000 7fd01739d640 -1 osd.2 0 log_to_monitors true 2024-02-05T04:57:09.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:10.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:10.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:10.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:57:10.078 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T04:57:10.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:10.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:57:10.439 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/683616416,v1:127.0.0.1:6819/683616416] [v2:127.0.0.1:6820/683616416,v1:127.0.0.1:6821/683616416] exists,up 736811a5-b2aa-4a25-9c44-e570a1b64129 2024-02-05T04:57:10.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:57:10.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:57:10.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:57:10.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:57:10.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-02-05T04:57:10.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:57:10.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:57:10.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T04:57:10.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:57:10.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-02-05T04:57:10.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:57:10.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:57:10.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:57:10.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:57:10.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-02-05T04:57:10.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-02-05T04:57:10.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:57:10.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:57:10.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:57:10.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:57:10.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:57:10.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:57:10.443 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:57:10.443 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:57:10.443 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T04:57:10.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T04:57:10.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T04:57:10.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:57:10.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:57:10.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:57:10.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:57:10.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:57:10.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:57:10.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:57:10.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:57:10.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:57:10.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:57:10.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-02-05T04:57:10.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-02-05T04:57:10.446 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:57:10.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c87d5483-9e0c-47ab-b984-a7964f4e6a0a 2024-02-05T04:57:10.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 c87d5483-9e0c-47ab-b984-a7964f4e6a0a' 2024-02-05T04:57:10.447 INFO:tasks.workunit.client.0.smithi148.stdout:add osd3 c87d5483-9e0c-47ab-b984-a7964f4e6a0a 2024-02-05T04:57:10.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:57:10.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCmasBlBPTnGxAAxDr9GGFtgS7C7+mcRm3N6Q== 2024-02-05T04:57:10.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCmasBlBPTnGxAAxDr9GGFtgS7C7+mcRm3N6Q=="}' 2024-02-05T04:57:10.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c87d5483-9e0c-47ab-b984-a7964f4e6a0a -i td/osd-recovery-scrub/3/new.json 2024-02-05T04:57:10.833 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T04:57:10.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-02-05T04:57:10.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCmasBlBPTnGxAAxDr9GGFtgS7C7+mcRm3N6Q== --osd-uuid c87d5483-9e0c-47ab-b984-a7964f4e6a0a 2024-02-05T04:57:10.889 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:10.883+0000 7fcc85905640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:10.909 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:10.903+0000 7fcc85905640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:10.910 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:10.907+0000 7fcc85905640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:10.911 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:10.907+0000 7fcc85905640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-05T04:57:10.911 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:10.907+0000 7fcc85905640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-05T04:57:10.911 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:10.907+0000 7fcc85905640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-05T04:57:10.976 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:10.971+0000 7fcc85905640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-02-05T04:57:17.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-02-05T04:57:17.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:57:17.288 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd3 key to auth repository 2024-02-05T04:57:17.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T04:57:17.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:57:17.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T04:57:17.664 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.3 2024-02-05T04:57:17.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-02-05T04:57:17.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:57:17.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:57:17.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:57:17.696 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:17.691+0000 7fc1e0ae2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:17.737 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:17.731+0000 7fc1e0ae2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:17.738 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:17.735+0000 7fc1e0ae2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:17.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T04:57:17.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:57:17.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T04:57:17.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:57:17.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:57:17.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:17.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:57:17.959 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T04:57:17.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:17.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T04:57:18.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:18.283 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:18.279+0000 7fc1e0ae2640 -1 Falling back to public interface 2024-02-05T04:57:19.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:19.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:19.272 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T04:57:19.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:57:19.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:19.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T04:57:19.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:20.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:20.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:20.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:57:20.577 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T04:57:20.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:20.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T04:57:20.814 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T04:57:20.807+0000 7fc1e0ae2640 -1 osd.3 0 log_to_monitors true 2024-02-05T04:57:20.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:21.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:21.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:21.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:57:21.908 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T04:57:21.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:21.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T04:57:22.224 INFO:tasks.workunit.client.0.smithi148.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2114704289,v1:127.0.0.1:6827/2114704289] [v2:127.0.0.1:6828/2114704289,v1:127.0.0.1:6829/2114704289] exists,up c87d5483-9e0c-47ab-b984-a7964f4e6a0a 2024-02-05T04:57:22.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:57:22.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:57:22.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:57:22.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-02-05T04:57:22.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-05T04:57:22.648 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'test' created 2024-02-05T04:57:22.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T04:57:23.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-02-05T04:57:23.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:57:23.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:57:23.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:57:23.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:57:23.671 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:57:23.671 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:57:23.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:57:23.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:57:23.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:57:23.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:57:23.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:57:23.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:57:23.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:57:23.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:57:23.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:57:24.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:57:24.211 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T04:57:24.211 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T04:57:24.211 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T04:57:24.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:57:24.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:57:24.212 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:57:24.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-02-05T04:57:24.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-02-05T04:57:24.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-02-05T04:57:24.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:57:24.352 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:57:24.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-02-05T04:57:24.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-02-05T04:57:24.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2024-02-05T04:57:24.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:57:24.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:57:24.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-02-05T04:57:24.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-02-05T04:57:24.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509445' 2024-02-05T04:57:24.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:57:24.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T04:57:24.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345922 2024-02-05T04:57:24.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345922 2024-02-05T04:57:24.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509445 3-85899345922' 2024-02-05T04:57:24.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:57:24.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-02-05T04:57:24.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:57:24.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:57:24.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-02-05T04:57:24.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:57:24.780 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836489 2024-02-05T04:57:24.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-02-05T04:57:24.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-02-05T04:57:24.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:57:25.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-02-05T04:57:25.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:57:25.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-02-05T04:57:25.079 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:57:25.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:57:25.081 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-02-05T04:57:25.081 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:57:25.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-02-05T04:57:25.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-02-05T04:57:25.083 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672967 2024-02-05T04:57:25.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:57:25.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-02-05T04:57:25.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:57:25.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-02-05T04:57:25.428 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:57:25.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:57:25.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-02-05T04:57:25.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:57:25.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-02-05T04:57:25.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-02-05T04:57:25.432 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509445 2024-02-05T04:57:25.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:57:25.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-02-05T04:57:25.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:57:25.745 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345922 2024-02-05T04:57:25.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:57:25.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T04:57:25.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345922 2024-02-05T04:57:25.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:57:25.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345922 2024-02-05T04:57:25.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-02-05T04:57:25.749 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.3 seq 85899345922 2024-02-05T04:57:25.750 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T04:57:26.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345922 -lt 85899345922 2024-02-05T04:57:26.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:57:26.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:57:26.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:57:26.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:57:26.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:57:26.413 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:57:26.413 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:57:26.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:57:26.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:57:26.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:57:26.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:57:26.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:57:26.716 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:57:26.716 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:57:26.716 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:57:27.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:57:27.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:57:27.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:57:27.090 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-02-05T04:57:27.090 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-02-05T04:57:27.091 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-02-05T04:57:27.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-02-05T04:57:27.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-02-05T04:57:27.680 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:57:27.681 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:57:23.710948+0000 0'0 24:10 [1] 1 [1] 1 0'0 2024-02-05T04:57:22.597657+0000 0'0 2024-02-05T04:57:22.597657+0000 0 0 periodic scrub scheduled @ 2024-02-06T06:10:00.741551+0000 0 0 2024-02-05T04:57:27.681 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T04:57:27.681 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:57:27.681 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T04:57:27.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.36362 bs=1M count=50 2024-02-05T04:57:27.854 INFO:tasks.workunit.client.0.smithi148.stderr:50+0 records in 2024-02-05T04:57:27.854 INFO:tasks.workunit.client.0.smithi148.stderr:50+0 records out 2024-02-05T04:57:27.854 INFO:tasks.workunit.client.0.smithi148.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.157458 s, 333 MB/s 2024-02-05T04:57:27.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-02-05T04:57:27.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:27.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.36362 2024-02-05T04:57:29.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:29.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.36362 2024-02-05T04:57:31.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:31.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.36362 2024-02-05T04:57:32.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:32.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.36362 2024-02-05T04:57:34.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:34.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.36362 2024-02-05T04:57:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.36362 2024-02-05T04:57:37.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:37.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.36362 2024-02-05T04:57:39.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:39.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.36362 2024-02-05T04:57:40.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:40.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.36362 2024-02-05T04:57:42.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:42.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.36362 2024-02-05T04:57:43.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:43.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.36362 2024-02-05T04:57:45.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:45.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.36362 2024-02-05T04:57:46.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:46.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.36362 2024-02-05T04:57:48.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:48.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.36362 2024-02-05T04:57:50.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:50.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.36362 2024-02-05T04:57:51.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:51.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.36362 2024-02-05T04:57:52.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:52.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.36362 2024-02-05T04:57:54.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:54.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.36362 2024-02-05T04:57:55.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:55.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.36362 2024-02-05T04:57:57.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:57.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.36362 2024-02-05T04:57:58.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:57:58.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.36362 2024-02-05T04:58:00.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:00.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.36362 2024-02-05T04:58:01.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:01.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.36362 2024-02-05T04:58:03.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:03.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.36362 2024-02-05T04:58:05.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:05.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.36362 2024-02-05T04:58:06.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:06.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.36362 2024-02-05T04:58:08.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:08.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.36362 2024-02-05T04:58:09.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:09.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.36362 2024-02-05T04:58:11.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:11.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.36362 2024-02-05T04:58:12.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:12.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.36362 2024-02-05T04:58:14.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:14.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.36362 2024-02-05T04:58:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.36362 2024-02-05T04:58:17.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:17.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.36362 2024-02-05T04:58:18.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:18.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.36362 2024-02-05T04:58:20.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:20.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.36362 2024-02-05T04:58:21.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:21.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.36362 2024-02-05T04:58:23.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:23.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.36362 2024-02-05T04:58:24.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:24.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.36362 2024-02-05T04:58:26.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:26.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.36362 2024-02-05T04:58:27.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:27.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.36362 2024-02-05T04:58:29.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:29.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.36362 2024-02-05T04:58:30.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:30.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.36362 2024-02-05T04:58:32.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:32.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.36362 2024-02-05T04:58:33.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:33.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.36362 2024-02-05T04:58:35.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:35.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.36362 2024-02-05T04:58:36.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:36.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.36362 2024-02-05T04:58:38.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:38.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.36362 2024-02-05T04:58:40.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:40.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.36362 2024-02-05T04:58:41.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:41.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.36362 2024-02-05T04:58:43.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:43.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.36362 2024-02-05T04:58:45.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:45.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.36362 2024-02-05T04:58:46.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:46.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.36362 2024-02-05T04:58:48.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:48.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.36362 2024-02-05T04:58:49.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:49.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.36362 2024-02-05T04:58:51.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:51.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.36362 2024-02-05T04:58:52.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:52.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.36362 2024-02-05T04:58:54.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:54.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.36362 2024-02-05T04:58:56.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:56.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.36362 2024-02-05T04:58:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.36362 2024-02-05T04:58:59.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:58:59.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.36362 2024-02-05T04:59:00.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:00.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.36362 2024-02-05T04:59:02.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:02.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.36362 2024-02-05T04:59:04.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:04.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.36362 2024-02-05T04:59:05.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:05.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.36362 2024-02-05T04:59:07.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:07.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.36362 2024-02-05T04:59:08.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:08.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.36362 2024-02-05T04:59:10.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:10.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.36362 2024-02-05T04:59:11.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:11.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.36362 2024-02-05T04:59:13.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:13.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.36362 2024-02-05T04:59:14.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:14.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.36362 2024-02-05T04:59:16.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:16.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.36362 2024-02-05T04:59:17.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:17.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.36362 2024-02-05T04:59:19.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:19.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.36362 2024-02-05T04:59:20.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:20.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.36362 2024-02-05T04:59:22.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:22.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.36362 2024-02-05T04:59:23.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:23.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.36362 2024-02-05T04:59:25.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:25.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.36362 2024-02-05T04:59:26.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:26.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.36362 2024-02-05T04:59:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.36362 2024-02-05T04:59:29.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:29.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.36362 2024-02-05T04:59:31.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:31.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.36362 2024-02-05T04:59:32.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:32.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.36362 2024-02-05T04:59:34.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:34.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.36362 2024-02-05T04:59:36.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:36.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.36362 2024-02-05T04:59:37.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:37.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.36362 2024-02-05T04:59:38.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:38.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.36362 2024-02-05T04:59:40.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:40.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.36362 2024-02-05T04:59:42.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:42.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.36362 2024-02-05T04:59:43.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:43.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.36362 2024-02-05T04:59:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.36362 2024-02-05T04:59:46.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:46.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.36362 2024-02-05T04:59:48.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:48.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.36362 2024-02-05T04:59:49.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:49.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.36362 2024-02-05T04:59:50.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:50.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.36362 2024-02-05T04:59:52.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:52.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.36362 2024-02-05T04:59:54.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:54.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.36362 2024-02-05T04:59:55.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:55.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.36362 2024-02-05T04:59:56.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:56.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.36362 2024-02-05T04:59:58.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:59:58.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.36362 2024-02-05T05:00:00.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T05:00:00.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.36362 2024-02-05T05:00:01.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.36362 2024-02-05T05:00:01.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-02-05T05:00:01.857 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 1 size to 4 2024-02-05T05:00:01.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-02-05T05:00:01.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-02-05T05:00:01.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-05T05:00:01.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-05T05:00:01.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-05T05:00:01.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-05T05:00:02.184 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:00:02.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-02-05T05:00:04.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-02-05T05:00:04.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-02-05T05:00:04.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-02-05T05:00:04.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-05T05:00:04.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-05T05:00:04.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-05T05:00:04.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-05T05:00:04.512 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:00:04.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-02-05T05:00:06.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-02-05T05:00:06.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-02-05T05:00:06.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-02-05T05:00:06.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-05T05:00:06.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-05T05:00:06.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-05T05:00:06.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-05T05:00:06.836 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:00:06.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-02-05T05:00:08.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2024-02-05T05:00:08.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2024-02-05T05:00:08.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2024-02-05T05:00:08.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-05T05:00:08.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-05T05:00:08.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-05T05:00:08.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-05T05:00:09.166 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:00:09.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-02-05T05:00:11.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 3 -eq 10 2024-02-05T05:00:11.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 3 + 1 2024-02-05T05:00:11.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=4 2024-02-05T05:00:11.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-05T05:00:11.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-05T05:00:11.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-05T05:00:11.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-05T05:00:11.946 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:00:11.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-02-05T05:00:11.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-02-05T05:00:11.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-02-05T05:00:12.260 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T05:00:12.260 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 100 0 297 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-02-05T05:00:04.578495+0000 24'1300 27:1326 [1,0,2,3] 1 [1,0] 1 0'0 2024-02-05T04:57:22.597657+0000 0'0 2024-02-05T04:57:22.597657+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:00:12.260 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T05:00:12.260 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T05:00:12.261 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:00:12.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-02-05T05:00:22.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-02-05T05:00:22.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:00:22.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:00:22.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:00:22.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:00:22.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:00:27.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:00:27.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-02-05T05:00:27.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress. Only high priority scrubs allowed.' 2024-02-05T05:00:27.601 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-02-05T05:00:27.602 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-02-05T05:00:27.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:00:27.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-02-05T05:00:27.606 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:05.432+0000 7f3f4d299640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.606 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:06.428+0000 7f3f4d299640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.607 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:09.384+0000 7f3f4d299640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.607 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:11.372+0000 7f3f4d299640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.607 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:12.408+0000 7f3f4d299640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.607 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:13.392+0000 7f3f4d299640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.607 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:15.364+0000 7f3f4d299640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.607 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:21.396+0000 7f3f4d299640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:00:27.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-02-05T05:00:27.623 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:09.884+0000 7fc825060640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.623 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:15.908+0000 7fc825060640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.623 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:20.912+0000 7fc825060640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.623 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:22.876+0000 7fc825060640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.623 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:23.904+0000 7fc825060640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.623 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:24.888+0000 7fc825060640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.623 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:25.908+0000 7fc825060640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:00:27.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-02-05T05:00:27.626 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:07.556+0000 7fd010351640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.627 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:10.644+0000 7fd010351640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.627 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:16.604+0000 7fd010351640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.627 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:17.576+0000 7fd010351640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.627 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:26.560+0000 7fd010351640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:00:27.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-02-05T05:00:27.630 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:07.500+0000 7fc1d9a96640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.630 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:08.508+0000 7fc1d9a96640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.630 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:12.580+0000 7fc1d9a96640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.630 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:14.588+0000 7fc1d9a96640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.630 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:16.600+0000 7fc1d9a96640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.630 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:22.656+0000 7fc1d9a96640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.630 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:23.608+0000 7fc1d9a96640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.630 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:00:25.612+0000 7fc1d9a96640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T05:00:27.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-02-05T05:00:27.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-02-05T05:00:27.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-02-05T05:00:27.631 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-02-05T05:00:27.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-02-05T05:00:27.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:00:27.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-02-05T05:00:27.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-02-05T05:00:27.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-02-05T05:00:27.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-02-05T05:00:27.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:00:27.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-02-05T05:00:27.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-02-05T05:00:27.653 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-02-05T05:00:27.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-02-05T05:00:27.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:00:27.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-02-05T05:00:27.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-02-05T05:00:27.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-02-05T05:00:27.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-02-05T05:00:27.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:00:27.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-02-05T05:00:27.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-02-05T05:00:27.662 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-02-05T05:00:27.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-02-05T05:00:27.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-02-05T05:00:27.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-02-05T05:00:27.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-02-05T05:00:27.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-02-05T05:00:27.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:00:27.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-02-05T05:00:27.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:00:27.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:00:27.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:00:27.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:00:27.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:00:27.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:00:27.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:00:27.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:00:27.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:00:27.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:00:27.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:00:27.673 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:00:27.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:00:27.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:00:27.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:00:27.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:00:27.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:00:27.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:00:27.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-02-05T05:00:27.700 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:00:27.700 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:00:27.700 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:00:27.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36362 2024-02-05T05:00:27.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:00:27.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:00:27.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-02-05T05:00:27.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-02-05T05:00:27.701 INFO:tasks.workunit.client.0.smithi148.stdout:TEST PASSED 2024-02-05T05:00:27.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-02-05T05:00:27.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-02-05T05:00:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-02-05T05:00:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-02-05T05:00:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2024-02-05T05:00:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.36362 2024-02-05T05:00:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2024-02-05T05:00:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2024-02-05T05:00:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2024-02-05T05:00:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-02-05T05:00:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-02-05T05:00:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-02-05T05:00:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-02-05T05:00:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:00:27.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-02-05T05:00:27.705 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:00:27.705 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:00:27.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:00:27.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:00:27.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:00:27.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:00:27.706 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:00:27.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:00:27.706 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:00:27.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:00:27.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:00:27.706 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:00:27.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:00:27.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:00:27.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:00:27.708 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:00:27.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:00:27.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:00:27.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-02-05T05:00:27.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:00:27.710 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:00:27.710 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:00:27.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36362 2024-02-05T05:00:27.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:00:27.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:00:27.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-02-05T05:00:27.712 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:00:27.712 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:00:27.712 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:00:27.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36362 2024-02-05T05:00:27.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:00:27.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:00:27.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:00:27.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-02-05T05:00:27.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T05:00:27.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-02-05T05:00:27.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:00:27.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:00:27.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:00:27.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-02-05T05:00:27.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T05:00:28.087 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:00:28.087 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:00:28.087 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:00:28.087 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:00:28.087 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:00:28.087 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:00:28.087 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:00:28.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T05:00:28.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:00:28.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:00:28.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:00:28.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:00:28.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:00:28.290 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:00:28.290 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:00:28.290 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:00:28.290 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:00:28.290 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:00:28.290 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:00:28.290 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:00:28.290 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36362/ceph-mon.a.asok 2024-02-05T05:00:28.291 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:00:28.291 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36362/ceph-mon.a.asok config get fsid 2024-02-05T05:00:28.405 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:00:28.405 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:00:28.405 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:00:28.405 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:00:28.406 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:00:28.406 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:00:28.406 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:00:28.407 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:00:28.407 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:00:28.407 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:00:28.407 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:00:28.407 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36362/ceph-mon.a.asok 2024-02-05T05:00:28.407 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:00:28.407 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36362/ceph-mon.a.asok config get mon_host 2024-02-05T05:00:28.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-02-05T05:00:28.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-02-05T05:00:28.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:00:28.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:00:28.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:00:28.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-02-05T05:00:28.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:00:29.363 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:00:29.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:00:29.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:00:29.364 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:00:29.364 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:00:29.364 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:00:29.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:00:29.365 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:00:29.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:00:29.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-02-05T05:00:29.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-02-05T05:00:29.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-02-05T05:00:29.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-02-05T05:00:29.401 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2024-02-05T05:00:29.402 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2024-02-05T05:00:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:00:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:00:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T05:00:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:00:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:00:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:00:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-02-05T05:00:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T05:00:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:00:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:00:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:00:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-02-05T05:00:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-02-05T05:00:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:00:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:00:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T05:00:29.407 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:00:29.407 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:00:29.407 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:00:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:00:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:00:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:00:29.407 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:00:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:00:29.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:00:29.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:00:29.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:00:29.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T05:00:29.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T05:00:29.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:00:29.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:00:29.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:00:29.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:00:29.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:00:29.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T05:00:29.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-02-05T05:00:29.408 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:00:29.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dba22832-3e92-4faf-b5b4-5df4bd57ad57 2024-02-05T05:00:29.409 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 dba22832-3e92-4faf-b5b4-5df4bd57ad57 2024-02-05T05:00:29.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dba22832-3e92-4faf-b5b4-5df4bd57ad57' 2024-02-05T05:00:29.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:00:29.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBta8BlISp4GRAAZf+Y8Ni/SQaS0zDvS59W0Q== 2024-02-05T05:00:29.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBta8BlISp4GRAAZf+Y8Ni/SQaS0zDvS59W0Q=="}' 2024-02-05T05:00:29.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dba22832-3e92-4faf-b5b4-5df4bd57ad57 -i td/osd-recovery-scrub/0/new.json 2024-02-05T05:00:29.676 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:00:29.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-02-05T05:00:29.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBta8BlISp4GRAAZf+Y8Ni/SQaS0zDvS59W0Q== --osd-uuid dba22832-3e92-4faf-b5b4-5df4bd57ad57 2024-02-05T05:00:29.730 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:29.728+0000 7f68e8f00640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:29.732 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:29.728+0000 7f68e8f00640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:29.733 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:29.732+0000 7f68e8f00640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:29.733 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:29.732+0000 7f68e8f00640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-05T05:00:29.734 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:29.732+0000 7f68e8f00640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-05T05:00:29.734 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:29.732+0000 7f68e8f00640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-05T05:00:29.819 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:29.816+0000 7f68e8f00640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-02-05T05:00:35.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-02-05T05:00:35.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:00:35.616 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:00:35.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:00:35.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:00:36.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:00:36.008 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:00:36.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:00:36.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:00:36.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:00:36.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:00:36.045 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:36.044+0000 7f9ce1c2d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:36.047 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:36.044+0000 7f9ce1c2d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:36.048 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:36.044+0000 7f9ce1c2d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:36.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:00:36.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:00:36.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:00:36.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:00:36.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:00:36.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:36.330 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:00:36.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:00:36.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:36.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:00:36.599 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:36.595+0000 7f9ce1c2d640 -1 Falling back to public interface 2024-02-05T05:00:36.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:37.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:37.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:37.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:00:37.628 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:00:37.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:37.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:00:37.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:38.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:38.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:38.937 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:00:38.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:00:38.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:38.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:00:39.136 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:39.131+0000 7f9ce1c2d640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:00:39.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:40.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:40.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:40.253 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:00:40.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:00:40.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:40.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:00:40.587 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3205500962,v1:127.0.0.1:6803/3205500962] [v2:127.0.0.1:6804/3205500962,v1:127.0.0.1:6805/3205500962] exists,up dba22832-3e92-4faf-b5b4-5df4bd57ad57 2024-02-05T05:00:40.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:00:40.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:00:40.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:00:40.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:00:40.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:00:40.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T05:00:40.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:00:40.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:00:40.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:00:40.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-02-05T05:00:40.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T05:00:40.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:00:40.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:00:40.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:00:40.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-02-05T05:00:40.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-02-05T05:00:40.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:00:40.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:00:40.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T05:00:40.591 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:00:40.591 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:00:40.591 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:00:40.591 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:00:40.591 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:00:40.591 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:00:40.592 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:00:40.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:00:40.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:00:40.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:00:40.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:00:40.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T05:00:40.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T05:00:40.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:00:40.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:00:40.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:00:40.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:00:40.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:00:40.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T05:00:40.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-02-05T05:00:40.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:00:40.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fca17335-ecdd-4c3c-868b-fc316c76e35f 2024-02-05T05:00:40.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fca17335-ecdd-4c3c-868b-fc316c76e35f' 2024-02-05T05:00:40.594 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 fca17335-ecdd-4c3c-868b-fc316c76e35f 2024-02-05T05:00:40.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:00:40.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB4a8BlRNCiJBAAJeAp/Ezo3+wbR3CW1AVyhQ== 2024-02-05T05:00:40.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB4a8BlRNCiJBAAJeAp/Ezo3+wbR3CW1AVyhQ=="}' 2024-02-05T05:00:40.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fca17335-ecdd-4c3c-868b-fc316c76e35f -i td/osd-recovery-scrub/1/new.json 2024-02-05T05:00:40.964 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:00:40.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-02-05T05:00:40.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQB4a8BlRNCiJBAAJeAp/Ezo3+wbR3CW1AVyhQ== --osd-uuid fca17335-ecdd-4c3c-868b-fc316c76e35f 2024-02-05T05:00:41.023 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:41.019+0000 7f251fc92640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:41.048 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:41.047+0000 7f251fc92640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:41.050 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:41.047+0000 7f251fc92640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:41.050 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:41.047+0000 7f251fc92640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-05T05:00:41.050 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:41.047+0000 7f251fc92640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-05T05:00:41.051 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:41.047+0000 7f251fc92640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-05T05:00:41.112 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:41.107+0000 7f251fc92640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-02-05T05:00:47.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-02-05T05:00:47.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:00:47.276 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:00:47.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:00:47.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:00:47.657 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:00:47.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:00:47.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:00:47.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:00:47.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:00:47.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:00:47.695 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:47.691+0000 7f9ac9ad2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:47.721 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:47.719+0000 7f9ac9ad2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:47.722 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:47.719+0000 7f9ac9ad2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:47.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:00:47.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:00:47.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:00:47.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:00:47.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:00:47.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:47.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:00:47.982 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:00:47.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:47.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:48.271 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:48.267+0000 7f9ac9ad2640 -1 Falling back to public interface 2024-02-05T05:00:48.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:49.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:49.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:49.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:00:49.303 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:00:49.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:49.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:49.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:50.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:50.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:50.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:00:50.610 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:00:50.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:50.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:50.880 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:50.875+0000 7f9ac9ad2640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:00:50.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:51.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:51.913 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:00:51.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:51.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:00:51.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:51.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:52.246 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1452261221,v1:127.0.0.1:6811/1452261221] [v2:127.0.0.1:6812/1452261221,v1:127.0.0.1:6813/1452261221] exists,up fca17335-ecdd-4c3c-868b-fc316c76e35f 2024-02-05T05:00:52.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:00:52.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:00:52.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:00:52.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:00:52.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:00:52.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T05:00:52.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:00:52.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:00:52.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:00:52.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-02-05T05:00:52.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T05:00:52.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:00:52.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:00:52.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:00:52.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-02-05T05:00:52.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-02-05T05:00:52.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:00:52.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:00:52.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T05:00:52.252 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:00:52.253 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:00:52.253 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:00:52.253 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:00:52.253 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:00:52.253 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:00:52.253 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:00:52.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:00:52.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:00:52.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:00:52.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:00:52.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T05:00:52.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T05:00:52.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:00:52.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:00:52.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:00:52.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:00:52.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:00:52.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T05:00:52.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-02-05T05:00:52.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:00:52.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=27e14d4a-9eed-4374-8899-b18d763a94d0 2024-02-05T05:00:52.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 27e14d4a-9eed-4374-8899-b18d763a94d0' 2024-02-05T05:00:52.255 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 27e14d4a-9eed-4374-8899-b18d763a94d0 2024-02-05T05:00:52.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:00:52.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCEa8Bl8tEIEBAAQJxxCEfTQs+lF1VtzOe0rQ== 2024-02-05T05:00:52.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCEa8Bl8tEIEBAAQJxxCEfTQs+lF1VtzOe0rQ=="}' 2024-02-05T05:00:52.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 27e14d4a-9eed-4374-8899-b18d763a94d0 -i td/osd-recovery-scrub/2/new.json 2024-02-05T05:00:52.637 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:00:52.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-02-05T05:00:52.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCEa8Bl8tEIEBAAQJxxCEfTQs+lF1VtzOe0rQ== --osd-uuid 27e14d4a-9eed-4374-8899-b18d763a94d0 2024-02-05T05:00:52.694 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:52.691+0000 7f5bfa06e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:52.721 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:52.719+0000 7f5bfa06e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:52.722 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:52.719+0000 7f5bfa06e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:52.723 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:52.719+0000 7f5bfa06e640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-05T05:00:52.723 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:52.719+0000 7f5bfa06e640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-05T05:00:52.723 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:52.719+0000 7f5bfa06e640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-05T05:00:52.794 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:52.791+0000 7f5bfa06e640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-02-05T05:00:58.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-02-05T05:00:58.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:00:58.592 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T05:00:58.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:00:58.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:00:58.981 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T05:00:58.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:00:58.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:00:58.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:00:58.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:00:58.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:00:59.019 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:59.015+0000 7f500e515640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:59.046 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:59.043+0000 7f500e515640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:59.047 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:59.043+0000 7f500e515640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:59.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:00:59.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:00:59.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:00:59.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:00:59.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:00:59.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:59.303 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:00:59.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:00:59.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:59.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:59.594 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:00:59.591+0000 7f500e515640 -1 Falling back to public interface 2024-02-05T05:00:59.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:00.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:00.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:00.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:01:00.630 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:01:00.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:00.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:01:00.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:01.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:01.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:01.949 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:01:01.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:01:01.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:01.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:01:02.173 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:02.171+0000 7f500e515640 -1 osd.2 0 log_to_monitors true 2024-02-05T05:01:02.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:03.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:03.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:03.277 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:01:03.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:01:03.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:03.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:01:03.597 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/124828745,v1:127.0.0.1:6819/124828745] [v2:127.0.0.1:6820/124828745,v1:127.0.0.1:6821/124828745] exists,up 27e14d4a-9eed-4374-8899-b18d763a94d0 2024-02-05T05:01:03.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:01:03.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:01:03.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:01:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:01:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:01:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T05:01:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:01:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T05:01:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:01:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-02-05T05:01:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T05:01:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:01:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:01:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:01:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-02-05T05:01:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-02-05T05:01:03.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:01:03.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:01:03.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T05:01:03.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:01:03.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:01:03.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:01:03.603 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:01:03.603 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:03.603 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:01:03.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:01:03.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:01:03.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:01:03.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:01:03.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:01:03.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T05:01:03.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T05:01:03.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:01:03.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:01:03.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:01:03.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:01:03.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:01:03.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T05:01:03.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-02-05T05:01:03.605 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:01:03.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3dad412d-c512-49a3-b43f-ca0e9e55ec93 2024-02-05T05:01:03.605 INFO:tasks.workunit.client.0.smithi148.stdout:add osd3 3dad412d-c512-49a3-b43f-ca0e9e55ec93 2024-02-05T05:01:03.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 3dad412d-c512-49a3-b43f-ca0e9e55ec93' 2024-02-05T05:01:03.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:01:03.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCPa8BlGfIJJRAAuqf2yT6Ptn2Hx7fHkjnsyQ== 2024-02-05T05:01:03.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCPa8BlGfIJJRAAuqf2yT6Ptn2Hx7fHkjnsyQ=="}' 2024-02-05T05:01:03.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3dad412d-c512-49a3-b43f-ca0e9e55ec93 -i td/osd-recovery-scrub/3/new.json 2024-02-05T05:01:03.959 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:01:03.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-02-05T05:01:03.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCPa8BlGfIJJRAAuqf2yT6Ptn2Hx7fHkjnsyQ== --osd-uuid 3dad412d-c512-49a3-b43f-ca0e9e55ec93 2024-02-05T05:01:04.016 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:04.011+0000 7fb9e6461640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:04.040 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:04.035+0000 7fb9e6461640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:04.041 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:04.039+0000 7fb9e6461640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:04.042 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:04.039+0000 7fb9e6461640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-05T05:01:04.042 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:04.039+0000 7fb9e6461640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-05T05:01:04.042 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:04.039+0000 7fb9e6461640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-05T05:01:04.136 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:04.131+0000 7fb9e6461640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-02-05T05:01:10.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-02-05T05:01:10.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:01:10.222 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd3 key to auth repository 2024-02-05T05:01:10.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T05:01:10.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:01:10.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T05:01:10.610 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.3 2024-02-05T05:01:10.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:01:10.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:01:10.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:01:10.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:01:10.647 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:10.643+0000 7f50ef018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:10.649 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:10.647+0000 7f50ef018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:10.650 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:10.647+0000 7f50ef018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:10.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T05:01:10.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:01:10.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:01:10.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:01:10.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:01:10.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:10.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:01:10.900 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:01:10.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:10.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:01:11.195 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:11.191+0000 7f50ef018640 -1 Falling back to public interface 2024-02-05T05:01:11.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:12.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:12.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:12.214 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:01:12.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:01:12.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:12.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:01:12.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:13.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:13.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:13.526 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:01:13.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:01:13.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:13.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:01:13.746 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:13.743+0000 7f50ef018640 -1 osd.3 0 log_to_monitors true 2024-02-05T05:01:13.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:14.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:14.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:14.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:01:14.850 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:01:14.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:14.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:01:15.180 INFO:tasks.workunit.client.0.smithi148.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/116071788,v1:127.0.0.1:6827/116071788] [v2:127.0.0.1:6828/116071788,v1:127.0.0.1:6829/116071788] exists,up 3dad412d-c512-49a3-b43f-ca0e9e55ec93 2024-02-05T05:01:15.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:01:15.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:01:15.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:01:15.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:01:15.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:01:15.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T05:01:15.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:01:15.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-02-05T05:01:15.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:01:15.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-02-05T05:01:15.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T05:01:15.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:01:15.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:01:15.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:01:15.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-02-05T05:01:15.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-02-05T05:01:15.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:01:15.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:01:15.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T05:01:15.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:01:15.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:01:15.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:01:15.184 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:01:15.184 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:15.185 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:01:15.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:01:15.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:01:15.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:01:15.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:01:15.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:01:15.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T05:01:15.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T05:01:15.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:01:15.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:01:15.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:01:15.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:01:15.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:01:15.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T05:01:15.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-02-05T05:01:15.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:01:15.186 INFO:tasks.workunit.client.0.smithi148.stdout:add osd4 70611a8a-4b16-4563-ae3d-8c46b580ac78 2024-02-05T05:01:15.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=70611a8a-4b16-4563-ae3d-8c46b580ac78 2024-02-05T05:01:15.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 70611a8a-4b16-4563-ae3d-8c46b580ac78' 2024-02-05T05:01:15.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:01:15.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCba8Bl2MkwDBAA1nb96rVt3O/SurxKpLDSmA== 2024-02-05T05:01:15.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCba8Bl2MkwDBAA1nb96rVt3O/SurxKpLDSmA=="}' 2024-02-05T05:01:15.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 70611a8a-4b16-4563-ae3d-8c46b580ac78 -i td/osd-recovery-scrub/4/new.json 2024-02-05T05:01:15.550 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:01:15.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-02-05T05:01:15.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCba8Bl2MkwDBAA1nb96rVt3O/SurxKpLDSmA== --osd-uuid 70611a8a-4b16-4563-ae3d-8c46b580ac78 2024-02-05T05:01:15.607 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:15.603+0000 7fdb6e6fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:15.630 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:15.627+0000 7fdb6e6fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:15.631 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:15.627+0000 7fdb6e6fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:15.632 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:15.627+0000 7fdb6e6fa640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-02-05T05:01:15.632 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:15.627+0000 7fdb6e6fa640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-02-05T05:01:15.632 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:15.627+0000 7fdb6e6fa640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-02-05T05:01:15.701 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:15.699+0000 7fdb6e6fa640 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-02-05T05:01:21.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-02-05T05:01:21.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:01:21.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-02-05T05:01:21.800 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd4 key to auth repository 2024-02-05T05:01:21.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:01:22.180 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.4 2024-02-05T05:01:22.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-02-05T05:01:22.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:01:22.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:01:22.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:01:22.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:01:22.216 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:22.215+0000 7fe59eae1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:22.244 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:22.239+0000 7fe59eae1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:22.245 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:22.243+0000 7fe59eae1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-02-05T05:01:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:01:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-02-05T05:01:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:01:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:01:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:22.513 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:01:22.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:01:22.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:22.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:01:22.794 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:22.791+0000 7fe59eae1640 -1 Falling back to public interface 2024-02-05T05:01:22.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:23.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:23.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:23.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:01:23.826 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:01:23.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:23.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:01:24.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:25.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:25.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:25.124 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:01:25.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:01:25.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:25.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:01:25.241 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:25.239+0000 7fe59eae1640 -1 osd.4 0 log_to_monitors true 2024-02-05T05:01:25.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:26.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:26.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:26.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:01:26.445 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:01:26.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:26.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:01:26.744 INFO:tasks.workunit.client.0.smithi148.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2289428246,v1:127.0.0.1:6835/2289428246] [v2:127.0.0.1:6836/2289428246,v1:127.0.0.1:6837/2289428246] exists,up 70611a8a-4b16-4563-ae3d-8c46b580ac78 2024-02-05T05:01:26.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:01:26.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:01:26.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:01:26.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:01:26.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:01:26.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T05:01:26.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:01:26.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:01:26.749 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:01:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T05:01:26.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-02-05T05:01:26.751 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:01:26.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0e6de5a0-6881-4976-8c38-cf58d877bf03 2024-02-05T05:01:26.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 0e6de5a0-6881-4976-8c38-cf58d877bf03' 2024-02-05T05:01:26.752 INFO:tasks.workunit.client.0.smithi148.stdout:add osd5 0e6de5a0-6881-4976-8c38-cf58d877bf03 2024-02-05T05:01:26.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:01:26.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCma8BlHiAELhAATUBwZ1A7k97p6AMegT6NKA== 2024-02-05T05:01:26.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCma8BlHiAELhAATUBwZ1A7k97p6AMegT6NKA=="}' 2024-02-05T05:01:26.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0e6de5a0-6881-4976-8c38-cf58d877bf03 -i td/osd-recovery-scrub/5/new.json 2024-02-05T05:01:27.142 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:01:27.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-02-05T05:01:27.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCma8BlHiAELhAATUBwZ1A7k97p6AMegT6NKA== --osd-uuid 0e6de5a0-6881-4976-8c38-cf58d877bf03 2024-02-05T05:01:27.203 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:27.199+0000 7fd816030640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:27.205 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:27.203+0000 7fd816030640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:27.206 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:27.203+0000 7fd816030640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:27.207 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:27.203+0000 7fd816030640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-02-05T05:01:27.207 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:27.203+0000 7fd816030640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-02-05T05:01:27.207 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:27.203+0000 7fd816030640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-02-05T05:01:27.293 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:27.291+0000 7fd816030640 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-02-05T05:01:33.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-02-05T05:01:33.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:01:33.211 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd5 key to auth repository 2024-02-05T05:01:33.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-02-05T05:01:33.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:01:33.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-02-05T05:01:33.597 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.5 2024-02-05T05:01:33.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:01:33.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:01:33.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:01:33.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:01:33.633 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:33.631+0000 7ff01b348640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:33.661 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:33.659+0000 7ff01b348640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:33.662 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:33.659+0000 7ff01b348640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:33.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-02-05T05:01:33.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:01:33.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:01:33.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:01:33.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:01:33.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:33.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:01:33.909 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:01:33.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:33.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:01:34.198 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:34.195+0000 7ff01b348640 -1 Falling back to public interface 2024-02-05T05:01:34.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:35.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:35.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:35.243 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:01:35.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:01:35.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:35.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:01:35.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:36.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:36.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:36.554 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:01:36.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:01:36.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:36.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:01:36.747 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:36.743+0000 7ff01b348640 -1 osd.5 0 log_to_monitors true 2024-02-05T05:01:36.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:37.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:37.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:37.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:01:37.953 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:01:37.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:37.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:01:38.271 INFO:tasks.workunit.client.0.smithi148.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1398589566,v1:127.0.0.1:6843/1398589566] [v2:127.0.0.1:6844/1398589566,v1:127.0.0.1:6845/1398589566] exists,up 0e6de5a0-6881-4976-8c38-cf58d877bf03 2024-02-05T05:01:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:01:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:01:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:01:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:01:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:01:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T05:01:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:01:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-02-05T05:01:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:01:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-02-05T05:01:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T05:01:38.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:01:38.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:01:38.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:01:38.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-02-05T05:01:38.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-02-05T05:01:38.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:01:38.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:01:38.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T05:01:38.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:01:38.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:01:38.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:01:38.276 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:01:38.276 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:38.276 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:01:38.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:01:38.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:01:38.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:01:38.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:01:38.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:01:38.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T05:01:38.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T05:01:38.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:01:38.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:01:38.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:01:38.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:01:38.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:01:38.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T05:01:38.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-02-05T05:01:38.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:01:38.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=282ff203-4786-41c3-9212-0385142e4eba 2024-02-05T05:01:38.278 INFO:tasks.workunit.client.0.smithi148.stdout:add osd6 282ff203-4786-41c3-9212-0385142e4eba 2024-02-05T05:01:38.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 282ff203-4786-41c3-9212-0385142e4eba' 2024-02-05T05:01:38.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:01:38.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCya8BllrmhERAA/GjMue5urxGwjdutwOf9UQ== 2024-02-05T05:01:38.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCya8BllrmhERAA/GjMue5urxGwjdutwOf9UQ=="}' 2024-02-05T05:01:38.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 282ff203-4786-41c3-9212-0385142e4eba -i td/osd-recovery-scrub/6/new.json 2024-02-05T05:01:38.669 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:01:38.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-02-05T05:01:38.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCya8BllrmhERAA/GjMue5urxGwjdutwOf9UQ== --osd-uuid 282ff203-4786-41c3-9212-0385142e4eba 2024-02-05T05:01:38.728 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:38.727+0000 7f9d755e3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:38.737 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:38.735+0000 7f9d755e3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:38.738 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:38.735+0000 7f9d755e3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:38.738 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:38.735+0000 7f9d755e3640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-02-05T05:01:38.738 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:38.735+0000 7f9d755e3640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-02-05T05:01:38.738 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:38.735+0000 7f9d755e3640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-02-05T05:01:38.801 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:38.799+0000 7f9d755e3640 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-02-05T05:01:44.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-02-05T05:01:44.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:01:44.998 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd6 key to auth repository 2024-02-05T05:01:44.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-02-05T05:01:44.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:01:45.380 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.6 2024-02-05T05:01:45.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-02-05T05:01:45.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:01:45.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:01:45.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:01:45.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:01:45.418 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:45.415+0000 7f0dfed0a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:45.440 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:45.439+0000 7f0dfed0a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:45.441 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:45.439+0000 7f0dfed0a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:45.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-02-05T05:01:45.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:01:45.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-02-05T05:01:45.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:01:45.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:01:45.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:45.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:01:45.698 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:01:45.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:45.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:01:46.002 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:45.999+0000 7f0dfed0a640 -1 Falling back to public interface 2024-02-05T05:01:46.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:47.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:47.028 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:01:47.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:01:47.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:47.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:01:47.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:48.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:48.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:48.336 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:01:48.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:01:48.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:48.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:01:48.614 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:48.611+0000 7f0dfed0a640 -1 osd.6 0 log_to_monitors true 2024-02-05T05:01:48.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:49.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:49.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:49.655 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:01:49.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:01:49.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:49.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:01:49.998 INFO:tasks.workunit.client.0.smithi148.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1923225943,v1:127.0.0.1:6851/1923225943] [v2:127.0.0.1:6852/1923225943,v1:127.0.0.1:6853/1923225943] exists,up 282ff203-4786-41c3-9212-0385142e4eba 2024-02-05T05:01:50.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:01:50.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:01:50.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:01:50.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:01:50.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:01:50.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T05:01:50.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:01:50.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-02-05T05:01:50.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:01:50.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-02-05T05:01:50.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T05:01:50.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:01:50.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:01:50.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:01:50.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-02-05T05:01:50.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-02-05T05:01:50.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:01:50.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:01:50.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T05:01:50.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:01:50.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:01:50.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:01:50.001 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:01:50.001 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:50.001 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:01:50.003 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:01:50.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' 2024-02-05T05:01:50.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:01:50.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:01:50.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:01:50.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T05:01:50.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T05:01:50.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:01:50.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:01:50.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:01:50.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:01:50.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:01:50.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T05:01:50.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-02-05T05:01:50.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:01:50.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=db9b54b2-ccbf-443f-978b-d7fab8071839 2024-02-05T05:01:50.004 INFO:tasks.workunit.client.0.smithi148.stdout:add osd7 db9b54b2-ccbf-443f-978b-d7fab8071839 2024-02-05T05:01:50.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 db9b54b2-ccbf-443f-978b-d7fab8071839' 2024-02-05T05:01:50.005 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:01:50.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC+a8BlizmXARAApglJg360ne94Pi18g1e06A== 2024-02-05T05:01:50.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC+a8BlizmXARAApglJg360ne94Pi18g1e06A=="}' 2024-02-05T05:01:50.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new db9b54b2-ccbf-443f-978b-d7fab8071839 -i td/osd-recovery-scrub/7/new.json 2024-02-05T05:01:50.376 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:01:50.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-02-05T05:01:50.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQC+a8BlizmXARAApglJg360ne94Pi18g1e06A== --osd-uuid db9b54b2-ccbf-443f-978b-d7fab8071839 2024-02-05T05:01:50.436 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:50.435+0000 7f0b58da8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:50.445 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:50.443+0000 7f0b58da8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:50.446 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:50.443+0000 7f0b58da8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:50.447 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:50.443+0000 7f0b58da8640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-02-05T05:01:50.447 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:50.443+0000 7f0b58da8640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-02-05T05:01:50.447 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:50.443+0000 7f0b58da8640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-02-05T05:01:50.507 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:50.503+0000 7f0b58da8640 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-02-05T05:01:56.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-02-05T05:01:56.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:01:56.629 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd7 key to auth repository 2024-02-05T05:01:56.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-02-05T05:01:56.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:01:57.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-02-05T05:01:57.004 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.7 2024-02-05T05:01:57.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=1f75fc59-ab7b-4840-9b39-0f165c6e90a6 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36362/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T05:01:57.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:01:57.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:01:57.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:01:57.042 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:57.039+0000 7fd042a3b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:57.071 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:57.067+0000 7fd042a3b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:57.073 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:57.071+0000 7fd042a3b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:57.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-02-05T05:01:57.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:01:57.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-02-05T05:01:57.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:01:57.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:01:57.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:57.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:01:57.339 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:01:57.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:57.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:01:57.625 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:01:57.623+0000 7fd042a3b640 -1 Falling back to public interface 2024-02-05T05:01:57.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:58.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:58.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:58.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:01:58.651 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:01:58.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:58.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:01:58.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:59.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:59.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:59.953 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:01:59.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:01:59.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:59.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:02:00.179 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:02:00.179+0000 7fd042a3b640 -1 osd.7 0 log_to_monitors true 2024-02-05T05:02:00.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:01.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:01.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:01.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:02:01.281 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:02:01.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:01.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:02:01.599 INFO:tasks.workunit.client.0.smithi148.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/879618588,v1:127.0.0.1:6859/879618588] [v2:127.0.0.1:6860/879618588,v1:127.0.0.1:6861/879618588] exists,up db9b54b2-ccbf-443f-978b-d7fab8071839 2024-02-05T05:02:01.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:02:01.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:02:01.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:02:01.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2024-02-05T05:02:01.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2024-02-05T05:02:02.165 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'test' created 2024-02-05T05:02:02.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:02:03.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2024-02-05T05:02:03.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:02:03.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:02:03.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:02:03.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:02:03.190 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:02:03.191 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:02:03.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:02:03.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:02:03.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:02:03.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:02:03.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:02:03.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:02:03.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:02:03.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:02:03.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:02:03.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:02:03.758 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:02:03.758 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:02:03.758 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:02:03.758 INFO:tasks.workunit.client.0.smithi148.stderr:4 2024-02-05T05:02:03.758 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:02:03.758 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:02:03.758 INFO:tasks.workunit.client.0.smithi148.stderr:7' 2024-02-05T05:02:03.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:02:03.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:02:03.758 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:02:04.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-02-05T05:02:04.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-02-05T05:02:04.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-02-05T05:02:04.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:02:04.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:02:04.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-02-05T05:02:04.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-02-05T05:02:04.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2024-02-05T05:02:04.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:02:04.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:02:04.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-02-05T05:02:04.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-02-05T05:02:04.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454' 2024-02-05T05:02:04.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:02:04.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:02:04.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345931 2024-02-05T05:02:04.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345931 2024-02-05T05:02:04.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931' 2024-02-05T05:02:04.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:02:04.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:02:04.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182410 2024-02-05T05:02:04.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182410 2024-02-05T05:02:04.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931 4-107374182410' 2024-02-05T05:02:04.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:02:04.845 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:02:04.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-02-05T05:02:04.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-02-05T05:02:04.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931 4-107374182410 5-128849018887' 2024-02-05T05:02:04.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:02:04.984 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:02:05.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855364 2024-02-05T05:02:05.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855364 2024-02-05T05:02:05.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931 4-107374182410 5-128849018887 6-150323855364' 2024-02-05T05:02:05.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:02:05.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:02:05.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-02-05T05:02:05.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-02-05T05:02:05.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931 4-107374182410 5-128849018887 6-150323855364 7-171798691842' 2024-02-05T05:02:05.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:02:05.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-02-05T05:02:05.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:02:05.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:02:05.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-02-05T05:02:05.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:02:05.274 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836498 2024-02-05T05:02:05.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-02-05T05:02:05.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-02-05T05:02:05.275 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:02:05.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-02-05T05:02:05.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:02:06.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:02:06.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:02:06.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836498 2024-02-05T05:02:06.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:02:06.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:02:06.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-02-05T05:02:06.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:02:06.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-02-05T05:02:06.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:02:06.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-02-05T05:02:06.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-02-05T05:02:06.893 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672976 2024-02-05T05:02:06.893 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:02:07.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-02-05T05:02:07.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:02:07.215 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-02-05T05:02:07.215 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:02:07.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:02:07.216 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-02-05T05:02:07.216 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:02:07.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-02-05T05:02:07.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-02-05T05:02:07.218 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509454 2024-02-05T05:02:07.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:02:07.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-02-05T05:02:07.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:02:07.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:02:07.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345931 2024-02-05T05:02:07.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:02:07.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345931 2024-02-05T05:02:07.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:02:07.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345931 2024-02-05T05:02:07.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-02-05T05:02:07.524 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.3 seq 85899345931 2024-02-05T05:02:07.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:02:07.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345932 -lt 85899345931 2024-02-05T05:02:07.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:02:07.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182410 2024-02-05T05:02:07.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:02:07.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:02:07.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182410 2024-02-05T05:02:07.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:02:07.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182410 2024-02-05T05:02:07.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2024-02-05T05:02:07.844 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.4 seq 107374182410 2024-02-05T05:02:07.845 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:02:08.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182411 -lt 107374182410 2024-02-05T05:02:08.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:02:08.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018887 2024-02-05T05:02:08.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:02:08.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:02:08.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018887 2024-02-05T05:02:08.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:02:08.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-02-05T05:02:08.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-02-05T05:02:08.154 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.5 seq 128849018887 2024-02-05T05:02:08.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:02:08.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2024-02-05T05:02:08.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:02:08.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855364 2024-02-05T05:02:08.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:02:08.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:02:08.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855364 2024-02-05T05:02:08.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:02:08.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855364 2024-02-05T05:02:08.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855364' 2024-02-05T05:02:08.483 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.6 seq 150323855364 2024-02-05T05:02:08.483 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:02:08.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855364 2024-02-05T05:02:08.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:02:08.774 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691842 2024-02-05T05:02:08.774 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:02:08.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:02:08.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691842 2024-02-05T05:02:08.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:02:08.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-02-05T05:02:08.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691842' 2024-02-05T05:02:08.777 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.7 seq 171798691842 2024-02-05T05:02:08.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:02:09.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691842 2024-02-05T05:02:09.092 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:02:09.092 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:02:09.093 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:02:09.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-02-05T05:02:09.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:02:09.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:02:09.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:02:09.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:02:09.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:02:09.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:02:09.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:02:09.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-02-05T05:02:09.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:02:09.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:02:09.808 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:02:10.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-02-05T05:02:10.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:02:10.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:02:10.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2024-02-05T05:02:10.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-02-05T05:02:10.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-02-05T05:02:10.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2024-02-05T05:02:10.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.36362 bs=1M count=50 2024-02-05T05:02:10.678 INFO:tasks.workunit.client.0.smithi148.stderr:50+0 records in 2024-02-05T05:02:10.678 INFO:tasks.workunit.client.0.smithi148.stderr:50+0 records out 2024-02-05T05:02:10.678 INFO:tasks.workunit.client.0.smithi148.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.15964 s, 328 MB/s 2024-02-05T05:02:10.679 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2024-02-05T05:02:10.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:10.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.36362 2024-02-05T05:02:12.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:12.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.36362 2024-02-05T05:02:13.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:13.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.36362 2024-02-05T05:02:15.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:15.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.36362 2024-02-05T05:02:16.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:16.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.36362 2024-02-05T05:02:18.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:18.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.36362 2024-02-05T05:02:19.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:19.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.36362 2024-02-05T05:02:21.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:21.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.36362 2024-02-05T05:02:22.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:22.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.36362 2024-02-05T05:02:24.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:24.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.36362 2024-02-05T05:02:25.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:25.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.36362 2024-02-05T05:02:27.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:27.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.36362 2024-02-05T05:02:28.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:28.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.36362 2024-02-05T05:02:30.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:30.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.36362 2024-02-05T05:02:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.36362 2024-02-05T05:02:33.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:33.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.36362 2024-02-05T05:02:35.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:35.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.36362 2024-02-05T05:02:36.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:36.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.36362 2024-02-05T05:02:38.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:38.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.36362 2024-02-05T05:02:39.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:39.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.36362 2024-02-05T05:02:41.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:41.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.36362 2024-02-05T05:02:42.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:42.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.36362 2024-02-05T05:02:44.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:44.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.36362 2024-02-05T05:02:46.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:46.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.36362 2024-02-05T05:02:47.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:47.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.36362 2024-02-05T05:02:49.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:49.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.36362 2024-02-05T05:02:50.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:50.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.36362 2024-02-05T05:02:52.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:52.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.36362 2024-02-05T05:02:53.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:53.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.36362 2024-02-05T05:02:55.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:55.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.36362 2024-02-05T05:02:56.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:56.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.36362 2024-02-05T05:02:58.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:58.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.36362 2024-02-05T05:02:59.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:02:59.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.36362 2024-02-05T05:03:01.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:03:01.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.36362 2024-02-05T05:03:02.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:03:02.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.36362 2024-02-05T05:03:04.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:03:04.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.36362 2024-02-05T05:03:05.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:03:05.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.36362 2024-02-05T05:03:07.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:03:07.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.36362 2024-02-05T05:03:09.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:03:09.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.36362 2024-02-05T05:03:10.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T05:03:10.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.36362 2024-02-05T05:03:12.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.36362 2024-02-05T05:03:12.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-02-05T05:03:12.613 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 1 size to 3 2024-02-05T05:03:12.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2024-02-05T05:03:12.939 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T05:03:12.940 INFO:tasks.workunit.client.0.smithi148.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-02-05T05:02:03.614298+0000 47'26 47:52 [5] 5 [5] 5 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.940 INFO:tasks.workunit.client.0.smithi148.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.612938+0000 47'13 47:38 [3] 3 [3] 3 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.940 INFO:tasks.workunit.client.0.smithi148.stdout:1.1d 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-02-05T05:02:03.824287+0000 47'26 47:41 [1] 1 [1] 1 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.940 INFO:tasks.workunit.client.0.smithi148.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.665558+0000 47'13 47:30 [1] 1 [1] 1 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.940 INFO:tasks.workunit.client.0.smithi148.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.665786+0000 47'13 47:30 [1] 1 [1] 1 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.940 INFO:tasks.workunit.client.0.smithi148.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T05:02:03.666044+0000 0'0 47:17 [1] 1 [1] 1 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.940 INFO:tasks.workunit.client.0.smithi148.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-02-05T05:02:03.781008+0000 47'26 47:43 [5] 5 [5] 5 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.940 INFO:tasks.workunit.client.0.smithi148.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.781080+0000 47'13 47:29 [6] 6 [6] 6 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.940 INFO:tasks.workunit.client.0.smithi148.stdout:1.6 1 0 0 0 0 37748736 0 0 9 0 9 active+clean 2024-02-05T05:02:03.780726+0000 47'9 47:26 [6] 6 [6] 6 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.940 INFO:tasks.workunit.client.0.smithi148.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-02-05T05:02:03.665521+0000 47'39 47:57 [4] 4 [4] 4 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.940 INFO:tasks.workunit.client.0.smithi148.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-02-05T05:02:03.613082+0000 47'52 47:70 [3] 3 [3] 3 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.941 INFO:tasks.workunit.client.0.smithi148.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-02-05T05:02:03.665505+0000 47'52 47:71 [4] 4 [4] 4 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.941 INFO:tasks.workunit.client.0.smithi148.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.210133+0000 47'13 47:32 [0] 0 [0] 0 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.941 INFO:tasks.workunit.client.0.smithi148.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T05:02:03.780758+0000 0'0 47:15 [3] 3 [3] 3 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.941 INFO:tasks.workunit.client.0.smithi148.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.613539+0000 47'13 47:39 [5] 5 [5] 5 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.941 INFO:tasks.workunit.client.0.smithi148.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.210153+0000 47'13 47:27 [0] 0 [0] 0 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.941 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.209308+0000 47'13 47:38 [7] 7 [7] 7 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.941 INFO:tasks.workunit.client.0.smithi148.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.050405+0000 47'13 47:28 [7] 7 [7] 7 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.941 INFO:tasks.workunit.client.0.smithi148.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.823407+0000 47'13 47:29 [1] 1 [1] 1 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.941 INFO:tasks.workunit.client.0.smithi148.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.612730+0000 47'13 47:27 [6] 6 [6] 6 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.941 INFO:tasks.workunit.client.0.smithi148.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.614009+0000 47'13 47:29 [5] 5 [5] 5 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.941 INFO:tasks.workunit.client.0.smithi148.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.210303+0000 47'13 47:28 [0] 0 [0] 0 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.941 INFO:tasks.workunit.client.0.smithi148.stdout:1.12 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-02-05T05:02:03.050828+0000 47'26 47:40 [0] 0 [0] 0 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.942 INFO:tasks.workunit.client.0.smithi148.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.665271+0000 47'13 47:27 [4] 4 [4] 4 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.942 INFO:tasks.workunit.client.0.smithi148.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T05:02:03.665491+0000 0'0 47:14 [2] 2 [2] 2 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.942 INFO:tasks.workunit.client.0.smithi148.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-02-05T05:02:03.823937+0000 47'39 47:54 [1] 1 [1] 1 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.942 INFO:tasks.workunit.client.0.smithi148.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T05:02:03.781265+0000 0'0 47:13 [6] 6 [6] 6 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.942 INFO:tasks.workunit.client.0.smithi148.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.209941+0000 47'13 47:27 [0] 0 [0] 0 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.942 INFO:tasks.workunit.client.0.smithi148.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T05:02:03.781156+0000 0'0 47:15 [5] 5 [5] 5 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.942 INFO:tasks.workunit.client.0.smithi148.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.612762+0000 47'13 47:27 [3] 3 [3] 3 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.942 INFO:tasks.workunit.client.0.smithi148.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T05:02:03.824341+0000 0'0 47:15 [1] 1 [1] 1 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.942 INFO:tasks.workunit.client.0.smithi148.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T05:02:03.210416+0000 47'13 47:27 [0] 0 [0] 0 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:12.942 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T05:03:12.942 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T05:03:12.943 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:03:12.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2024-02-05T05:03:13.242 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T05:03:13.242 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_recovery_max_active": "0" 2024-02-05T05:03:13.242 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T05:03:13.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2024-02-05T05:03:13.472 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T05:03:13.473 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_recovery_max_active_hdd": "3" 2024-02-05T05:03:13.473 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T05:03:13.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2024-02-05T05:03:13.708 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T05:03:13.709 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_recovery_max_active_ssd": "10" 2024-02-05T05:03:13.709 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T05:03:13.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2024-02-05T05:03:13.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-05T05:03:13.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-05T05:03:13.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-05T05:03:13.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-05T05:03:13.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-05T05:03:14.124 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:03:14.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-02-05T05:03:14.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-05T05:03:16.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2024-02-05T05:03:16.145 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2024-02-05T05:03:16.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2024-02-05T05:03:16.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-05T05:03:16.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-05T05:03:16.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-05T05:03:16.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-05T05:03:16.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-05T05:03:16.453 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:03:16.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-02-05T05:03:16.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-05T05:03:18.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2024-02-05T05:03:18.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2024-02-05T05:03:18.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2024-02-05T05:03:18.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-05T05:03:18.477 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-05T05:03:18.478 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-05T05:03:18.478 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-05T05:03:18.478 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-05T05:03:18.773 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:03:18.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 3 -ge 2 2024-02-05T05:03:18.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:288: TEST_recovery_scrub_2: break 2024-02-05T05:03:18.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:298: TEST_recovery_scrub_2: ceph pg dump pgs 2024-02-05T05:03:19.080 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T05:03:19.080 INFO:tasks.workunit.client.0.smithi148.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 peering 2024-02-05T05:03:13.004308+0000 47'26 48:54 [5,0,2] 5 [5,0,2] 5 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.081 INFO:tasks.workunit.client.0.smithi148.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.600944+0000 47'13 50:48 [3,0,7] 3 [3,0,7] 3 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.081 INFO:tasks.workunit.client.0.smithi148.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-02-05T05:03:14.428176+0000 47'26 50:50 [1,3,0] 1 [1,3,0] 1 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.081 INFO:tasks.workunit.client.0.smithi148.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.598665+0000 47'13 50:39 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.081 INFO:tasks.workunit.client.0.smithi148.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.427345+0000 47'13 50:40 [1,6,2] 1 [1,6,2] 1 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.081 INFO:tasks.workunit.client.0.smithi148.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T05:03:14.183338+0000 0'0 50:29 [1,4,5] 1 [1,4,5] 1 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:19.081 INFO:tasks.workunit.client.0.smithi148.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 peering 2024-02-05T05:03:13.005630+0000 47'26 47:44 [5,4,3] 5 [5,4,3] 5 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.081 INFO:tasks.workunit.client.0.smithi148.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.596086+0000 47'13 50:38 [6,7,3] 6 [6,7,3] 6 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.081 INFO:tasks.workunit.client.0.smithi148.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.771060+0000 47'13 50:40 [6,7,3] 6 [6,7,3] 6 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.081 INFO:tasks.workunit.client.0.smithi148.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-02-05T05:03:14.594692+0000 47'39 50:67 [4,2,6] 4 [4,2,6] 4 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.081 INFO:tasks.workunit.client.0.smithi148.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-02-05T05:03:14.595562+0000 47'52 50:80 [3,0,6] 3 [3,0,6] 3 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.081 INFO:tasks.workunit.client.0.smithi148.stdout:1.3 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-02-05T05:03:14.771526+0000 47'52 50:80 [4,1,2] 4 [4,1,2] 4 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.082 INFO:tasks.workunit.client.0.smithi148.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.597836+0000 47'13 50:42 [0,7,1] 0 [0,7,1] 0 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.082 INFO:tasks.workunit.client.0.smithi148.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T05:03:14.263043+0000 0'0 50:27 [3,0,4] 3 [3,0,4] 3 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:19.082 INFO:tasks.workunit.client.0.smithi148.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 peering 2024-02-05T05:03:13.009671+0000 47'13 48:41 [5,0,4] 5 [5,0,4] 5 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.082 INFO:tasks.workunit.client.0.smithi148.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.674626+0000 47'13 50:36 [0,3,4] 0 [0,3,4] 0 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.082 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.264104+0000 47'13 50:48 [7,0,2] 7 [7,0,2] 7 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.082 INFO:tasks.workunit.client.0.smithi148.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-02-05T05:03:14.335252+0000 47'13 50:38 [7,4,2] 7 [7,4,2] 7 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.082 INFO:tasks.workunit.client.0.smithi148.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.432636+0000 47'13 50:38 [1,4,2] 1 [1,4,2] 1 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.082 INFO:tasks.workunit.client.0.smithi148.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.770872+0000 47'13 50:36 [6,0,3] 6 [6,0,3] 6 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.082 INFO:tasks.workunit.client.0.smithi148.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 peering 2024-02-05T05:03:13.006372+0000 47'13 47:30 [5,7,6] 5 [5,7,6] 5 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.082 INFO:tasks.workunit.client.0.smithi148.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.430274+0000 47'13 50:37 [0,1,3] 0 [0,1,3] 0 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.082 INFO:tasks.workunit.client.0.smithi148.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 active+recovering+degraded 2024-02-05T05:03:14.266690+0000 47'26 50:51 [0,1,6] 0 [0,1,6] 0 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.082 INFO:tasks.workunit.client.0.smithi148.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.427630+0000 47'13 50:36 [4,6,3] 4 [4,6,3] 4 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.083 INFO:tasks.workunit.client.0.smithi148.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 peering 2024-02-05T05:03:12.835982+0000 0'0 47:15 [2,0,4] 2 [2,0,4] 2 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.083 INFO:tasks.workunit.client.0.smithi148.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 active+recovering+degraded 2024-02-05T05:03:14.266674+0000 47'39 50:64 [1,5,3] 1 [1,5,3] 1 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.083 INFO:tasks.workunit.client.0.smithi148.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T05:03:14.428800+0000 0'0 50:24 [6,0,7] 6 [6,0,7] 6 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:19.083 INFO:tasks.workunit.client.0.smithi148.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.509485+0000 47'13 50:36 [0,4,5] 0 [0,4,5] 0 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.083 INFO:tasks.workunit.client.0.smithi148.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 peering 2024-02-05T05:03:13.003579+0000 0'0 47:16 [5,2,3] 5 [5,2,3] 5 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.083 INFO:tasks.workunit.client.0.smithi148.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.601783+0000 47'13 50:36 [3,0,4] 3 [3,0,4] 3 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.083 INFO:tasks.workunit.client.0.smithi148.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T05:03:14.430223+0000 0'0 50:26 [1,7,4] 1 [1,7,4] 1 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 periodic scrub scheduled @ 2024-02-06T05:02:02.024198+0000 0 0 2024-02-05T05:03:19.083 INFO:tasks.workunit.client.0.smithi148.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T05:03:14.429985+0000 47'13 50:36 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-05T05:02:02.024198+0000 0'0 2024-02-05T05:02:02.024198+0000 0 0 no scrub is scheduled 0 0 2024-02-05T05:03:19.083 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T05:03:19.083 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T05:03:19.083 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:03:19.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pids= 2024-02-05T05:03:19.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: recov_scrub_count=0 2024-02-05T05:03:19.098 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: expr 32 - 1 2024-02-05T05:03:19.099 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: seq 0 31 2024-02-05T05:03:19.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 0 2024-02-05T05:03:19.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2024-02-05T05:03:19.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57840"' 2024-02-05T05:03:19.103 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57840' 2024-02-05T05:03:19.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.0 2024-02-05T05:03:19.103 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 1 2024-02-05T05:03:19.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57843: /' 2024-02-05T05:03:19.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2024-02-05T05:03:19.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57845"' 2024-02-05T05:03:19.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57845' 2024-02-05T05:03:19.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1 2024-02-05T05:03:19.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57850: /' 2024-02-05T05:03:19.105 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 2 2024-02-05T05:03:19.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2024-02-05T05:03:19.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57853"' 2024-02-05T05:03:19.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57853' 2024-02-05T05:03:19.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 3 2024-02-05T05:03:19.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.2 2024-02-05T05:03:19.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57857: /' 2024-02-05T05:03:19.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2024-02-05T05:03:19.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57859"' 2024-02-05T05:03:19.107 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57859' 2024-02-05T05:03:19.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.107 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 4 2024-02-05T05:03:19.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57864: /' 2024-02-05T05:03:19.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2024-02-05T05:03:19.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57865"' 2024-02-05T05:03:19.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57865' 2024-02-05T05:03:19.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.3 2024-02-05T05:03:19.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 5 2024-02-05T05:03:19.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2024-02-05T05:03:19.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57872"' 2024-02-05T05:03:19.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57872' 2024-02-05T05:03:19.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.4 2024-02-05T05:03:19.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.5 2024-02-05T05:03:19.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57876: /' 2024-02-05T05:03:19.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57868: /' 2024-02-05T05:03:19.132 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 6 2024-02-05T05:03:19.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2024-02-05T05:03:19.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57882"' 2024-02-05T05:03:19.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57882' 2024-02-05T05:03:19.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 7 2024-02-05T05:03:19.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2024-02-05T05:03:19.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57885"' 2024-02-05T05:03:19.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57885' 2024-02-05T05:03:19.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.6 2024-02-05T05:03:19.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57889: /' 2024-02-05T05:03:19.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.7 2024-02-05T05:03:19.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57886: /' 2024-02-05T05:03:19.151 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 8 2024-02-05T05:03:19.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2024-02-05T05:03:19.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57894"' 2024-02-05T05:03:19.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57894' 2024-02-05T05:03:19.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.8 2024-02-05T05:03:19.162 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 9 2024-02-05T05:03:19.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2024-02-05T05:03:19.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57901"' 2024-02-05T05:03:19.164 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57901' 2024-02-05T05:03:19.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57897: /' 2024-02-05T05:03:19.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 10 2024-02-05T05:03:19.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2024-02-05T05:03:19.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57905"' 2024-02-05T05:03:19.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57905' 2024-02-05T05:03:19.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.9 2024-02-05T05:03:19.194 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 11 2024-02-05T05:03:19.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2024-02-05T05:03:19.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57910"' 2024-02-05T05:03:19.196 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57910' 2024-02-05T05:03:19.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57908: /' 2024-02-05T05:03:19.200 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 12 2024-02-05T05:03:19.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.a 2024-02-05T05:03:19.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57913: /' 2024-02-05T05:03:19.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2024-02-05T05:03:19.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57916"' 2024-02-05T05:03:19.211 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57916' 2024-02-05T05:03:19.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.b 2024-02-05T05:03:19.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57917: /' 2024-02-05T05:03:19.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 13 2024-02-05T05:03:19.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2024-02-05T05:03:19.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57926"' 2024-02-05T05:03:19.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57926' 2024-02-05T05:03:19.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57922: /' 2024-02-05T05:03:19.235 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 14 2024-02-05T05:03:19.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.c 2024-02-05T05:03:19.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2024-02-05T05:03:19.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57933"' 2024-02-05T05:03:19.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57933' 2024-02-05T05:03:19.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57932: /' 2024-02-05T05:03:19.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 15 2024-02-05T05:03:19.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2024-02-05T05:03:19.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57940"' 2024-02-05T05:03:19.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57940' 2024-02-05T05:03:19.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.d 2024-02-05T05:03:19.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.e 2024-02-05T05:03:19.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57939: /' 2024-02-05T05:03:19.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57949: /' 2024-02-05T05:03:19.286 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 16 2024-02-05T05:03:19.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2024-02-05T05:03:19.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57951"' 2024-02-05T05:03:19.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57951' 2024-02-05T05:03:19.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 17 2024-02-05T05:03:19.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2024-02-05T05:03:19.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57953"' 2024-02-05T05:03:19.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57953' 2024-02-05T05:03:19.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.f 2024-02-05T05:03:19.296 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 18 2024-02-05T05:03:19.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2024-02-05T05:03:19.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57962"' 2024-02-05T05:03:19.304 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57962' 2024-02-05T05:03:19.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57957: /' 2024-02-05T05:03:19.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.11 2024-02-05T05:03:19.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57961: /' 2024-02-05T05:03:19.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.12 2024-02-05T05:03:19.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.10 2024-02-05T05:03:19.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 19 2024-02-05T05:03:19.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2024-02-05T05:03:19.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57966: /' 2024-02-05T05:03:19.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57975"' 2024-02-05T05:03:19.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57975' 2024-02-05T05:03:19.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.359 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 20 2024-02-05T05:03:19.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2024-02-05T05:03:19.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57988"' 2024-02-05T05:03:19.360 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57988' 2024-02-05T05:03:19.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.13 2024-02-05T05:03:19.372 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 21 2024-02-05T05:03:19.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2024-02-05T05:03:19.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 57998"' 2024-02-05T05:03:19.374 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 57998' 2024-02-05T05:03:19.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/57987: /' 2024-02-05T05:03:19.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 22 2024-02-05T05:03:19.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58001: /' 2024-02-05T05:03:19.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2024-02-05T05:03:19.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58005"' 2024-02-05T05:03:19.390 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58005' 2024-02-05T05:03:19.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.15 2024-02-05T05:03:19.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58003: /' 2024-02-05T05:03:19.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.14 2024-02-05T05:03:19.407 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 23 2024-02-05T05:03:19.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2024-02-05T05:03:19.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58014"' 2024-02-05T05:03:19.408 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58014' 2024-02-05T05:03:19.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 24 2024-02-05T05:03:19.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2024-02-05T05:03:19.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58024"' 2024-02-05T05:03:19.428 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58024' 2024-02-05T05:03:19.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58019: /' 2024-02-05T05:03:19.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.16 2024-02-05T05:03:19.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58023: /' 2024-02-05T05:03:19.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.17 2024-02-05T05:03:19.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 25 2024-02-05T05:03:19.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2024-02-05T05:03:19.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58033"' 2024-02-05T05:03:19.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58033' 2024-02-05T05:03:19.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58028: /' 2024-02-05T05:03:19.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.18 2024-02-05T05:03:19.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 26 2024-02-05T05:03:19.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2024-02-05T05:03:19.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58043"' 2024-02-05T05:03:19.456 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58043' 2024-02-05T05:03:19.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58042: /' 2024-02-05T05:03:19.478 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 27 2024-02-05T05:03:19.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.19 2024-02-05T05:03:19.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2024-02-05T05:03:19.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58063"' 2024-02-05T05:03:19.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58063' 2024-02-05T05:03:19.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1a 2024-02-05T05:03:19.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58061: /' 2024-02-05T05:03:19.504 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 28 2024-02-05T05:03:19.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2024-02-05T05:03:19.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58078"' 2024-02-05T05:03:19.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58078' 2024-02-05T05:03:19.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58077: /' 2024-02-05T05:03:19.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1b 2024-02-05T05:03:19.526 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 29 2024-02-05T05:03:19.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2024-02-05T05:03:19.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58097"' 2024-02-05T05:03:19.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58097' 2024-02-05T05:03:19.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1c 2024-02-05T05:03:19.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58089: /' 2024-02-05T05:03:19.551 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 30 2024-02-05T05:03:19.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2024-02-05T05:03:19.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58121"' 2024-02-05T05:03:19.552 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58121' 2024-02-05T05:03:19.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:03:19.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1d 2024-02-05T05:03:19.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58104: /' 2024-02-05T05:03:19.571 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 31 2024-02-05T05:03:19.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2024-02-05T05:03:19.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:03:19.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:03:19.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58133"' 2024-02-05T05:03:19.572 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58133' 2024-02-05T05:03:19.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:306: TEST_recovery_scrub_2: wait_background_check pids 2024-02-05T05:03:19.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: pids=' 57840 57845 57853 57859 57865 57872 57882 57885 57894 57901 57905 57910 57916 57926 57933 57940 57951 57953 57962 57975 57988 57998 58005 58014 58024 58033 58043 58063 58078 58097 58121 58133' 2024-02-05T05:03:19.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: return_code=0 2024-02-05T05:03:19.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:19.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57840 2024-02-05T05:03:19.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58132: /' 2024-02-05T05:03:19.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1e 2024-02-05T05:03:19.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58146: /' 2024-02-05T05:03:19.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1f 2024-02-05T05:03:25.734 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.12 2024-02-05T05:03:25.735 INFO:tasks.workunit.client.0.smithi148.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.12 2024-02-05T05:03:25.735 INFO:tasks.workunit.client.0.smithi148.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-02-05T05:03:25.735 INFO:tasks.workunit.client.0.smithi148.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:25.735 INFO:tasks.workunit.client.0.smithi148.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:25.735 INFO:tasks.workunit.client.0.smithi148.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-02-05T05:03:25.735 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:25.735 INFO:tasks.workunit.client.0.smithi148.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.12 query 2024-02-05T05:03:25.735 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:25.735 INFO:tasks.workunit.client.0.smithi148.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:25.735 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:25.736 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:25.736 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.12 2024-02-05T05:03:25.736 INFO:tasks.workunit.client.0.smithi148.stderr:57966: instructing pg 1.12 on osd.0 to scrub 2024-02-05T05:03:25.736 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:25.736 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2024-02-05T05:03:25.736 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:25.736 INFO:tasks.workunit.client.0.smithi148.stderr:57966: dumped pgs 2024-02-05T05:03:25.736 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-02-05T05:03:25.736 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.12 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:25.736 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.12 2024-02-05T05:03:25.736 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:25.736 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:25.736 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:25.737 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:25.737 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:25.737 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:25.737 INFO:tasks.workunit.client.0.smithi148.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2024-02-05T05:03:25.737 INFO:tasks.workunit.client.0.smithi148.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-02-05T05:03:25.737 INFO:tasks.workunit.client.0.smithi148.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:25.737 INFO:tasks.workunit.client.0.smithi148.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:26.470 INFO:tasks.workunit.client.0.smithi148.stderr:57966: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-02-05T05:03:26.471 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:26.471 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:26.471 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-02-05T05:03:26.471 INFO:tasks.workunit.client.0.smithi148.stderr:57966: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-02-05T05:03:26.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-02-05T05:03:26.615 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.d 2024-02-05T05:03:26.615 INFO:tasks.workunit.client.0.smithi148.stderr:57932: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.d 2024-02-05T05:03:26.615 INFO:tasks.workunit.client.0.smithi148.stderr:57932: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-02-05T05:03:26.615 INFO:tasks.workunit.client.0.smithi148.stderr:57932: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:26.615 INFO:tasks.workunit.client.0.smithi148.stderr:57932: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-02-05T05:03:26.615 INFO:tasks.workunit.client.0.smithi148.stderr:57932: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:26.615 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:26.615 INFO:tasks.workunit.client.0.smithi148.stderr:57932: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:26.616 INFO:tasks.workunit.client.0.smithi148.stderr:57932: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.d query 2024-02-05T05:03:26.616 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:26.616 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:26.616 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:26.616 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.d 2024-02-05T05:03:26.616 INFO:tasks.workunit.client.0.smithi148.stderr:57932: instructing pg 1.d on osd.7 to scrub 2024-02-05T05:03:26.616 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2024-02-05T05:03:26.616 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:26.616 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:26.616 INFO:tasks.workunit.client.0.smithi148.stderr:57932: dumped pgs 2024-02-05T05:03:26.617 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-02-05T05:03:26.617 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.d 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:26.617 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.d 2024-02-05T05:03:26.617 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:26.617 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:26.617 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:26.617 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:26.617 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:26.617 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:26.617 INFO:tasks.workunit.client.0.smithi148.stderr:57932: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2024-02-05T05:03:26.617 INFO:tasks.workunit.client.0.smithi148.stderr:57932: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-02-05T05:03:26.617 INFO:tasks.workunit.client.0.smithi148.stderr:57932: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:26.618 INFO:tasks.workunit.client.0.smithi148.stderr:57932: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-02-05T05:03:26.951 INFO:tasks.workunit.client.0.smithi148.stderr:57932: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.8 2024-02-05T05:03:26.951 INFO:tasks.workunit.client.0.smithi148.stderr:57897: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.8 2024-02-05T05:03:26.951 INFO:tasks.workunit.client.0.smithi148.stderr:57897: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-02-05T05:03:26.951 INFO:tasks.workunit.client.0.smithi148.stderr:57897: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:26.951 INFO:tasks.workunit.client.0.smithi148.stderr:57897: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-02-05T05:03:26.951 INFO:tasks.workunit.client.0.smithi148.stderr:57897: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:26.951 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:26.952 INFO:tasks.workunit.client.0.smithi148.stderr:57897: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:26.952 INFO:tasks.workunit.client.0.smithi148.stderr:57897: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.8 query 2024-02-05T05:03:26.952 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:26.952 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:26.952 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:26.952 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.8 2024-02-05T05:03:26.952 INFO:tasks.workunit.client.0.smithi148.stderr:57897: instructing pg 1.8 on osd.5 to scrub 2024-02-05T05:03:26.952 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:26.952 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:26.952 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2024-02-05T05:03:26.952 INFO:tasks.workunit.client.0.smithi148.stderr:57897: dumped pgs 2024-02-05T05:03:26.952 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.8 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:26.952 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.8 2024-02-05T05:03:26.953 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:26.953 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:26.953 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:26.953 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:26.953 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:26.953 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:26.953 INFO:tasks.workunit.client.0.smithi148.stderr:57897: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2024-02-05T05:03:26.953 INFO:tasks.workunit.client.0.smithi148.stderr:57897: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-02-05T05:03:26.953 INFO:tasks.workunit.client.0.smithi148.stderr:57897: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:26.953 INFO:tasks.workunit.client.0.smithi148.stderr:57897: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-02-05T05:03:26.953 INFO:tasks.workunit.client.0.smithi148.stderr:57897: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:26.953 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:26.953 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:26.954 INFO:tasks.workunit.client.0.smithi148.stderr:57897: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:26.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:27.395 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.5 2024-02-05T05:03:27.395 INFO:tasks.workunit.client.0.smithi148.stderr:57876: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.5 2024-02-05T05:03:27.395 INFO:tasks.workunit.client.0.smithi148.stderr:57876: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-02-05T05:03:27.395 INFO:tasks.workunit.client.0.smithi148.stderr:57876: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.395 INFO:tasks.workunit.client.0.smithi148.stderr:57876: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.396 INFO:tasks.workunit.client.0.smithi148.stderr:57876: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-02-05T05:03:27.396 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.396 INFO:tasks.workunit.client.0.smithi148.stderr:57876: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.5 query 2024-02-05T05:03:27.396 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:27.396 INFO:tasks.workunit.client.0.smithi148.stderr:57876: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:27.396 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:27.396 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:27.396 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.5 2024-02-05T05:03:27.396 INFO:tasks.workunit.client.0.smithi148.stderr:57876: instructing pg 1.5 on osd.4 to scrub 2024-02-05T05:03:27.396 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2024-02-05T05:03:27.396 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:27.396 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:27.396 INFO:tasks.workunit.client.0.smithi148.stderr:57876: dumped pgs 2024-02-05T05:03:27.397 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.5 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.397 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.5 2024-02-05T05:03:27.397 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.397 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:27.397 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:27.397 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:27.397 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:27.397 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:27.397 INFO:tasks.workunit.client.0.smithi148.stderr:57876: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-02-05T05:03:27.397 INFO:tasks.workunit.client.0.smithi148.stderr:57876: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-02-05T05:03:27.398 INFO:tasks.workunit.client.0.smithi148.stderr:57876: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.398 INFO:tasks.workunit.client.0.smithi148.stderr:57876: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.398 INFO:tasks.workunit.client.0.smithi148.stderr:57876: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-02-05T05:03:27.398 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:27.398 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:27.398 INFO:tasks.workunit.client.0.smithi148.stderr:57876: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:27.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:27.432 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.2 2024-02-05T05:03:27.432 INFO:tasks.workunit.client.0.smithi148.stderr:57857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.2 2024-02-05T05:03:27.432 INFO:tasks.workunit.client.0.smithi148.stderr:57857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-02-05T05:03:27.432 INFO:tasks.workunit.client.0.smithi148.stderr:57857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.432 INFO:tasks.workunit.client.0.smithi148.stderr:57857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-02-05T05:03:27.432 INFO:tasks.workunit.client.0.smithi148.stderr:57857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.432 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.433 INFO:tasks.workunit.client.0.smithi148.stderr:57857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:27.433 INFO:tasks.workunit.client.0.smithi148.stderr:57857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.2 query 2024-02-05T05:03:27.433 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:27.433 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:27.433 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:27.433 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.2 2024-02-05T05:03:27.433 INFO:tasks.workunit.client.0.smithi148.stderr:57857: instructing pg 1.2 on osd.0 to scrub 2024-02-05T05:03:27.433 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2024-02-05T05:03:27.433 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:27.434 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:27.434 INFO:tasks.workunit.client.0.smithi148.stderr:57857: dumped pgs 2024-02-05T05:03:27.434 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.2 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.434 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.2 2024-02-05T05:03:27.434 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.434 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:27.434 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:27.434 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:27.435 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:27.435 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:27.435 INFO:tasks.workunit.client.0.smithi148.stderr:57857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-02-05T05:03:27.435 INFO:tasks.workunit.client.0.smithi148.stderr:57857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-02-05T05:03:27.435 INFO:tasks.workunit.client.0.smithi148.stderr:57857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.435 INFO:tasks.workunit.client.0.smithi148.stderr:57857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.435 INFO:tasks.workunit.client.0.smithi148.stderr:57857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-02-05T05:03:27.435 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:27.435 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:27.436 INFO:tasks.workunit.client.0.smithi148.stderr:57857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:27.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:27.548 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.6 2024-02-05T05:03:27.549 INFO:tasks.workunit.client.0.smithi148.stderr:57886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.6 2024-02-05T05:03:27.549 INFO:tasks.workunit.client.0.smithi148.stderr:57886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-02-05T05:03:27.549 INFO:tasks.workunit.client.0.smithi148.stderr:57886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.549 INFO:tasks.workunit.client.0.smithi148.stderr:57886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.549 INFO:tasks.workunit.client.0.smithi148.stderr:57886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-02-05T05:03:27.549 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.549 INFO:tasks.workunit.client.0.smithi148.stderr:57886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:27.549 INFO:tasks.workunit.client.0.smithi148.stderr:57886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.6 query 2024-02-05T05:03:27.549 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:27.549 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:27.549 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:27.549 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.6 2024-02-05T05:03:27.549 INFO:tasks.workunit.client.0.smithi148.stderr:57886: instructing pg 1.6 on osd.6 to scrub 2024-02-05T05:03:27.550 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2024-02-05T05:03:27.550 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:27.550 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:27.550 INFO:tasks.workunit.client.0.smithi148.stderr:57886: dumped pgs 2024-02-05T05:03:27.550 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.6 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.550 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.6 2024-02-05T05:03:27.550 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.550 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:27.550 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:27.550 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:27.550 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:27.550 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:27.550 INFO:tasks.workunit.client.0.smithi148.stderr:57886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-02-05T05:03:27.550 INFO:tasks.workunit.client.0.smithi148.stderr:57886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-02-05T05:03:27.551 INFO:tasks.workunit.client.0.smithi148.stderr:57886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.551 INFO:tasks.workunit.client.0.smithi148.stderr:57886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.551 INFO:tasks.workunit.client.0.smithi148.stderr:57886: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-02-05T05:03:27.551 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:27.551 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:27.551 INFO:tasks.workunit.client.0.smithi148.stderr:57886: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:27.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:27.624 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.a 2024-02-05T05:03:27.624 INFO:tasks.workunit.client.0.smithi148.stderr:57913: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.a 2024-02-05T05:03:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:57913: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-02-05T05:03:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:57913: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:57913: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-02-05T05:03:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:57913: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:57913: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:57913: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.a query 2024-02-05T05:03:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.a 2024-02-05T05:03:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:57913: instructing pg 1.a on osd.1 to scrub 2024-02-05T05:03:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2024-02-05T05:03:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:57913: dumped pgs 2024-02-05T05:03:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.a 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.a 2024-02-05T05:03:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:57913: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2024-02-05T05:03:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:57913: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-02-05T05:03:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:57913: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:57913: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.627 INFO:tasks.workunit.client.0.smithi148.stderr:57913: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-02-05T05:03:27.627 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:27.627 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:27.627 INFO:tasks.workunit.client.0.smithi148.stderr:57913: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:27.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:27.688 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1 2024-02-05T05:03:27.688 INFO:tasks.workunit.client.0.smithi148.stderr:57850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1 2024-02-05T05:03:27.688 INFO:tasks.workunit.client.0.smithi148.stderr:57850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:03:27.688 INFO:tasks.workunit.client.0.smithi148.stderr:57850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.688 INFO:tasks.workunit.client.0.smithi148.stderr:57850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.688 INFO:tasks.workunit.client.0.smithi148.stderr:57850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:03:27.689 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.689 INFO:tasks.workunit.client.0.smithi148.stderr:57850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:27.689 INFO:tasks.workunit.client.0.smithi148.stderr:57850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1 query 2024-02-05T05:03:27.689 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:27.689 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:27.689 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:27.689 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1 2024-02-05T05:03:27.690 INFO:tasks.workunit.client.0.smithi148.stderr:57850: instructing pg 1.1 on osd.5 to scrub 2024-02-05T05:03:27.690 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:27.690 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2024-02-05T05:03:27.690 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:27.690 INFO:tasks.workunit.client.0.smithi148.stderr:57850: dumped pgs 2024-02-05T05:03:27.690 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.690 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1 2024-02-05T05:03:27.690 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.691 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:27.691 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:27.691 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:27.691 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:27.691 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:27.691 INFO:tasks.workunit.client.0.smithi148.stderr:57850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-02-05T05:03:27.691 INFO:tasks.workunit.client.0.smithi148.stderr:57850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:03:27.691 INFO:tasks.workunit.client.0.smithi148.stderr:57850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.692 INFO:tasks.workunit.client.0.smithi148.stderr:57850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.692 INFO:tasks.workunit.client.0.smithi148.stderr:57850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:03:27.692 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:27.692 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:27.692 INFO:tasks.workunit.client.0.smithi148.stderr:57850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:27.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:27.709 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.7 2024-02-05T05:03:27.709 INFO:tasks.workunit.client.0.smithi148.stderr:57889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.7 2024-02-05T05:03:27.709 INFO:tasks.workunit.client.0.smithi148.stderr:57889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-02-05T05:03:27.709 INFO:tasks.workunit.client.0.smithi148.stderr:57889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.709 INFO:tasks.workunit.client.0.smithi148.stderr:57889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.709 INFO:tasks.workunit.client.0.smithi148.stderr:57889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-02-05T05:03:27.710 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.710 INFO:tasks.workunit.client.0.smithi148.stderr:57889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:27.710 INFO:tasks.workunit.client.0.smithi148.stderr:57889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.7 query 2024-02-05T05:03:27.710 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:27.710 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:27.710 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:27.710 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.7 2024-02-05T05:03:27.710 INFO:tasks.workunit.client.0.smithi148.stderr:57889: instructing pg 1.7 on osd.6 to scrub 2024-02-05T05:03:27.711 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2024-02-05T05:03:27.711 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:27.711 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:27.711 INFO:tasks.workunit.client.0.smithi148.stderr:57889: dumped pgs 2024-02-05T05:03:27.711 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.7 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.711 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.7 2024-02-05T05:03:27.711 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.712 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:27.712 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:27.712 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:27.712 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:27.712 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:27.712 INFO:tasks.workunit.client.0.smithi148.stderr:57889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-02-05T05:03:27.712 INFO:tasks.workunit.client.0.smithi148.stderr:57889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-02-05T05:03:27.712 INFO:tasks.workunit.client.0.smithi148.stderr:57889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.713 INFO:tasks.workunit.client.0.smithi148.stderr:57889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.713 INFO:tasks.workunit.client.0.smithi148.stderr:57889: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-02-05T05:03:27.713 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:27.713 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:27.713 INFO:tasks.workunit.client.0.smithi148.stderr:57889: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:27.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:27.738 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.11 2024-02-05T05:03:27.738 INFO:tasks.workunit.client.0.smithi148.stderr:57961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.11 2024-02-05T05:03:27.738 INFO:tasks.workunit.client.0.smithi148.stderr:57961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-02-05T05:03:27.738 INFO:tasks.workunit.client.0.smithi148.stderr:57961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.738 INFO:tasks.workunit.client.0.smithi148.stderr:57961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.738 INFO:tasks.workunit.client.0.smithi148.stderr:57961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-02-05T05:03:27.738 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.738 INFO:tasks.workunit.client.0.smithi148.stderr:57961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.11 query 2024-02-05T05:03:27.738 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:27.738 INFO:tasks.workunit.client.0.smithi148.stderr:57961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:27.738 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:27.738 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:27.739 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.11 2024-02-05T05:03:27.739 INFO:tasks.workunit.client.0.smithi148.stderr:57961: instructing pg 1.11 on osd.0 to scrub 2024-02-05T05:03:27.739 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:27.739 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2024-02-05T05:03:27.739 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:27.739 INFO:tasks.workunit.client.0.smithi148.stderr:57961: dumped pgs 2024-02-05T05:03:27.739 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.11 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.739 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.11 2024-02-05T05:03:27.739 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.739 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:27.739 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:27.739 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:27.739 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:27.739 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:27.740 INFO:tasks.workunit.client.0.smithi148.stderr:57961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2024-02-05T05:03:27.740 INFO:tasks.workunit.client.0.smithi148.stderr:57961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-02-05T05:03:27.740 INFO:tasks.workunit.client.0.smithi148.stderr:57961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.740 INFO:tasks.workunit.client.0.smithi148.stderr:57961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-02-05T05:03:27.740 INFO:tasks.workunit.client.0.smithi148.stderr:57961: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.740 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:27.740 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:27.740 INFO:tasks.workunit.client.0.smithi148.stderr:57961: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:27.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:27.877 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.19 2024-02-05T05:03:27.877 INFO:tasks.workunit.client.0.smithi148.stderr:58042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.19 2024-02-05T05:03:27.877 INFO:tasks.workunit.client.0.smithi148.stderr:58042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-02-05T05:03:27.877 INFO:tasks.workunit.client.0.smithi148.stderr:58042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.877 INFO:tasks.workunit.client.0.smithi148.stderr:58042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.877 INFO:tasks.workunit.client.0.smithi148.stderr:58042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-02-05T05:03:27.877 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.878 INFO:tasks.workunit.client.0.smithi148.stderr:58042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:27.878 INFO:tasks.workunit.client.0.smithi148.stderr:58042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.19 query 2024-02-05T05:03:27.878 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:27.878 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:27.878 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:27.878 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.19 2024-02-05T05:03:27.878 INFO:tasks.workunit.client.0.smithi148.stderr:58042: instructing pg 1.19 on osd.3 to scrub 2024-02-05T05:03:27.878 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:27.878 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2024-02-05T05:03:27.878 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:27.878 INFO:tasks.workunit.client.0.smithi148.stderr:58042: dumped pgs 2024-02-05T05:03:27.878 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.19 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.878 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.19 2024-02-05T05:03:27.879 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.879 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:27.879 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:27.879 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:27.879 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:27.879 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:27.879 INFO:tasks.workunit.client.0.smithi148.stderr:58042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2024-02-05T05:03:27.879 INFO:tasks.workunit.client.0.smithi148.stderr:58042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-02-05T05:03:27.879 INFO:tasks.workunit.client.0.smithi148.stderr:58042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.879 INFO:tasks.workunit.client.0.smithi148.stderr:58042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.879 INFO:tasks.workunit.client.0.smithi148.stderr:58042: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-02-05T05:03:27.879 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:27.879 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:27.879 INFO:tasks.workunit.client.0.smithi148.stderr:58042: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:27.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:27.881 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.0 2024-02-05T05:03:27.881 INFO:tasks.workunit.client.0.smithi148.stderr:57843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.0 2024-02-05T05:03:27.881 INFO:tasks.workunit.client.0.smithi148.stderr:57843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:03:27.882 INFO:tasks.workunit.client.0.smithi148.stderr:57843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.882 INFO:tasks.workunit.client.0.smithi148.stderr:57843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.882 INFO:tasks.workunit.client.0.smithi148.stderr:57843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:03:27.882 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.882 INFO:tasks.workunit.client.0.smithi148.stderr:57843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.0 query 2024-02-05T05:03:27.882 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:27.882 INFO:tasks.workunit.client.0.smithi148.stderr:57843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:27.882 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:27.882 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:27.882 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.0 2024-02-05T05:03:27.882 INFO:tasks.workunit.client.0.smithi148.stderr:57843: instructing pg 1.0 on osd.7 to scrub 2024-02-05T05:03:27.882 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2024-02-05T05:03:27.882 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:27.882 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: dumped pgs 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.0 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.0 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:03:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:27.884 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:27.884 INFO:tasks.workunit.client.0.smithi148.stderr:57843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:27.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:27.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:27.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:27.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:27.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:27.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57845 2024-02-05T05:03:27.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:27.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:27.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:27.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:27.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57853 2024-02-05T05:03:27.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:27.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:27.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:27.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:27.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57859 2024-02-05T05:03:28.042 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.15 2024-02-05T05:03:28.043 INFO:tasks.workunit.client.0.smithi148.stderr:58003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.15 2024-02-05T05:03:28.043 INFO:tasks.workunit.client.0.smithi148.stderr:58003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-02-05T05:03:28.043 INFO:tasks.workunit.client.0.smithi148.stderr:58003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.043 INFO:tasks.workunit.client.0.smithi148.stderr:58003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.043 INFO:tasks.workunit.client.0.smithi148.stderr:58003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-02-05T05:03:28.043 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.043 INFO:tasks.workunit.client.0.smithi148.stderr:58003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:28.043 INFO:tasks.workunit.client.0.smithi148.stderr:58003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.15 query 2024-02-05T05:03:28.043 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:28.043 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:28.043 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:28.043 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.15 2024-02-05T05:03:28.043 INFO:tasks.workunit.client.0.smithi148.stderr:58003: instructing pg 1.15 on osd.1 to scrub 2024-02-05T05:03:28.044 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:28.044 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:28.044 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2024-02-05T05:03:28.044 INFO:tasks.workunit.client.0.smithi148.stderr:58003: dumped pgs 2024-02-05T05:03:28.044 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-02-05T05:03:28.044 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.15 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.044 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.15 2024-02-05T05:03:28.044 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.044 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:28.044 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:28.044 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:28.044 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:28.044 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:28.045 INFO:tasks.workunit.client.0.smithi148.stderr:58003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2024-02-05T05:03:28.045 INFO:tasks.workunit.client.0.smithi148.stderr:58003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-02-05T05:03:28.045 INFO:tasks.workunit.client.0.smithi148.stderr:58003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.045 INFO:tasks.workunit.client.0.smithi148.stderr:58003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-02-05T05:03:28.230 INFO:tasks.workunit.client.0.smithi148.stderr:58003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: g58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.17 2024-02-05T05:03:28.230 INFO:tasks.workunit.client.0.smithi148.stderr:58019: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.17 2024-02-05T05:03:28.230 INFO:tasks.workunit.client.0.smithi148.stderr:58019: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-02-05T05:03:28.230 INFO:tasks.workunit.client.0.smithi148.stderr:58019: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.231 INFO:tasks.workunit.client.0.smithi148.stderr:58019: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.231 INFO:tasks.workunit.client.0.smithi148.stderr:58019: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-02-05T05:03:28.231 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.231 INFO:tasks.workunit.client.0.smithi148.stderr:58019: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:28.231 INFO:tasks.workunit.client.0.smithi148.stderr:58019: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.17 query 2024-02-05T05:03:28.231 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:28.231 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:28.231 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:28.231 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.17 2024-02-05T05:03:28.231 INFO:tasks.workunit.client.0.smithi148.stderr:58019: instructing pg 1.17 on osd.0 to scrub 2024-02-05T05:03:28.231 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:28.231 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2024-02-05T05:03:28.231 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:28.231 INFO:tasks.workunit.client.0.smithi148.stderr:58019: dumped pgs 2024-02-05T05:03:28.232 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.17 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.232 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.17 2024-02-05T05:03:28.232 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.232 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:28.232 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:28.232 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:28.232 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:28.232 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:28.232 INFO:tasks.workunit.client.0.smithi148.stderr:58019: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2024-02-05T05:03:28.232 INFO:tasks.workunit.client.0.smithi148.stderr:58019: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-02-05T05:03:28.232 INFO:tasks.workunit.client.0.smithi148.stderr:58019: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.232 INFO:tasks.workunit.client.0.smithi148.stderr:58019: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.232 INFO:tasks.workunit.client.0.smithi148.stderr:58019: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-02-05T05:03:28.232 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:28.233 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:28.233 INFO:tasks.workunit.client.0.smithi148.stderr:58019: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:28.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:28.239 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.4 2024-02-05T05:03:28.239 INFO:tasks.workunit.client.0.smithi148.stderr:57868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.4 2024-02-05T05:03:28.239 INFO:tasks.workunit.client.0.smithi148.stderr:57868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-02-05T05:03:28.239 INFO:tasks.workunit.client.0.smithi148.stderr:57868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.239 INFO:tasks.workunit.client.0.smithi148.stderr:57868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.239 INFO:tasks.workunit.client.0.smithi148.stderr:57868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-02-05T05:03:28.239 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.239 INFO:tasks.workunit.client.0.smithi148.stderr:57868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.4 query 2024-02-05T05:03:28.239 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:28.239 INFO:tasks.workunit.client.0.smithi148.stderr:57868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:28.239 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:28.239 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.4 2024-02-05T05:03:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:57868: instructing pg 1.4 on osd.3 to scrub 2024-02-05T05:03:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2024-02-05T05:03:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:57868: dumped pgs 2024-02-05T05:03:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.4 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.4 2024-02-05T05:03:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:28.241 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:28.241 INFO:tasks.workunit.client.0.smithi148.stderr:57868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-02-05T05:03:28.241 INFO:tasks.workunit.client.0.smithi148.stderr:57868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-02-05T05:03:28.241 INFO:tasks.workunit.client.0.smithi148.stderr:57868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.241 INFO:tasks.workunit.client.0.smithi148.stderr:57868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-02-05T05:03:28.241 INFO:tasks.workunit.client.0.smithi148.stderr:57868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.241 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:28.241 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:28.241 INFO:tasks.workunit.client.0.smithi148.stderr:57868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:28.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:28.600 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.c 2024-02-05T05:03:28.600 INFO:tasks.workunit.client.0.smithi148.stderr:57922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.c 2024-02-05T05:03:28.600 INFO:tasks.workunit.client.0.smithi148.stderr:57922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-02-05T05:03:28.600 INFO:tasks.workunit.client.0.smithi148.stderr:57922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.600 INFO:tasks.workunit.client.0.smithi148.stderr:57922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.600 INFO:tasks.workunit.client.0.smithi148.stderr:57922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-02-05T05:03:28.600 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.600 INFO:tasks.workunit.client.0.smithi148.stderr:57922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.c query 2024-02-05T05:03:28.600 INFO:tasks.workunit.client.0.smithi148.stderr:57922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:28.601 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:28.601 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:28.601 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:28.601 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.c 2024-02-05T05:03:28.601 INFO:tasks.workunit.client.0.smithi148.stderr:57922: instructing pg 1.c on osd.3 to scrub 2024-02-05T05:03:28.601 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2024-02-05T05:03:28.601 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:28.601 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:28.601 INFO:tasks.workunit.client.0.smithi148.stderr:57922: dumped pgs 2024-02-05T05:03:28.601 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.c 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.601 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.c 2024-02-05T05:03:28.601 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.601 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:28.602 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:28.602 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:28.602 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:28.602 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:28.602 INFO:tasks.workunit.client.0.smithi148.stderr:57922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2024-02-05T05:03:28.602 INFO:tasks.workunit.client.0.smithi148.stderr:57922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-02-05T05:03:28.602 INFO:tasks.workunit.client.0.smithi148.stderr:57922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.602 INFO:tasks.workunit.client.0.smithi148.stderr:57922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.602 INFO:tasks.workunit.client.0.smithi148.stderr:57922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-02-05T05:03:28.602 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:03:25.071012+0000 '>' '' 2024-02-05T05:03:28.602 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:28.602 INFO:tasks.workunit.client.0.smithi148.stderr:57922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:28.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:28.611 INFO:tasks.workunit.client.0.smithi148.stderr:_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.611 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:28.612 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:28.612 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-02-05T05:03:28.612 INFO:tasks.workunit.client.0.smithi148.stderr:57932: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-02-05T05:03:28.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-02-05T05:03:28.621 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.b 2024-02-05T05:03:28.621 INFO:tasks.workunit.client.0.smithi148.stderr:57917: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.b 2024-02-05T05:03:28.621 INFO:tasks.workunit.client.0.smithi148.stderr:57917: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-02-05T05:03:28.622 INFO:tasks.workunit.client.0.smithi148.stderr:57917: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.622 INFO:tasks.workunit.client.0.smithi148.stderr:57917: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.622 INFO:tasks.workunit.client.0.smithi148.stderr:57917: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-02-05T05:03:28.622 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.622 INFO:tasks.workunit.client.0.smithi148.stderr:57917: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:28.622 INFO:tasks.workunit.client.0.smithi148.stderr:57917: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.b query 2024-02-05T05:03:28.622 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:28.622 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:28.622 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:28.622 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.b 2024-02-05T05:03:28.622 INFO:tasks.workunit.client.0.smithi148.stderr:57917: instructing pg 1.b on osd.1 to scrub 2024-02-05T05:03:28.622 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:28.622 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2024-02-05T05:03:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:57917: dumped pgs 2024-02-05T05:03:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.b 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.b 2024-02-05T05:03:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:57917: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2024-02-05T05:03:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:57917: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-02-05T05:03:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:57917: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:57917: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-02-05T05:03:28.624 INFO:tasks.workunit.client.0.smithi148.stderr:57917: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.624 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:28.624 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:28.624 INFO:tasks.workunit.client.0.smithi148.stderr:57917: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:28.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:28.651 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1b 2024-02-05T05:03:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:58077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1b 2024-02-05T05:03:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:58077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-02-05T05:03:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:58077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:58077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:58077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-02-05T05:03:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:58077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:58077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1b query 2024-02-05T05:03:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1b 2024-02-05T05:03:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:58077: instructing pg 1.1b on osd.0 to scrub 2024-02-05T05:03:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2024-02-05T05:03:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:58077: dumped pgs 2024-02-05T05:03:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1b 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1b 2024-02-05T05:03:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:28.654 INFO:tasks.workunit.client.0.smithi148.stderr:58077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2024-02-05T05:03:28.654 INFO:tasks.workunit.client.0.smithi148.stderr:58077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-02-05T05:03:28.654 INFO:tasks.workunit.client.0.smithi148.stderr:58077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.654 INFO:tasks.workunit.client.0.smithi148.stderr:58077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.654 INFO:tasks.workunit.client.0.smithi148.stderr:58077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-02-05T05:03:28.654 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:28.654 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:28.654 INFO:tasks.workunit.client.0.smithi148.stderr:58077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:28.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:28.671 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.3 2024-02-05T05:03:28.671 INFO:tasks.workunit.client.0.smithi148.stderr:57864: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.3 2024-02-05T05:03:28.672 INFO:tasks.workunit.client.0.smithi148.stderr:57864: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:03:28.672 INFO:tasks.workunit.client.0.smithi148.stderr:57864: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.672 INFO:tasks.workunit.client.0.smithi148.stderr:57864: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:03:28.672 INFO:tasks.workunit.client.0.smithi148.stderr:57864: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.672 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.672 INFO:tasks.workunit.client.0.smithi148.stderr:57864: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:28.672 INFO:tasks.workunit.client.0.smithi148.stderr:57864: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.3 query 2024-02-05T05:03:28.672 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:28.672 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:28.672 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:28.672 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.3 2024-02-05T05:03:28.672 INFO:tasks.workunit.client.0.smithi148.stderr:57864: instructing pg 1.3 on osd.4 to scrub 2024-02-05T05:03:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2024-02-05T05:03:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:57864: dumped pgs 2024-02-05T05:03:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.3 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.3 2024-02-05T05:03:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:57864: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:03:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:57864: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:03:28.674 INFO:tasks.workunit.client.0.smithi148.stderr:57864: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.674 INFO:tasks.workunit.client.0.smithi148.stderr:57864: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.674 INFO:tasks.workunit.client.0.smithi148.stderr:57864: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:03:28.674 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:28.674 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:28.674 INFO:tasks.workunit.client.0.smithi148.stderr:57864: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:28.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:28.677 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.16 2024-02-05T05:03:28.677 INFO:tasks.workunit.client.0.smithi148.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.16 2024-02-05T05:03:28.677 INFO:tasks.workunit.client.0.smithi148.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-02-05T05:03:28.677 INFO:tasks.workunit.client.0.smithi148.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.677 INFO:tasks.workunit.client.0.smithi148.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-02-05T05:03:28.677 INFO:tasks.workunit.client.0.smithi148.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.677 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.677 INFO:tasks.workunit.client.0.smithi148.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:28.677 INFO:tasks.workunit.client.0.smithi148.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.16 query 2024-02-05T05:03:28.678 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:28.678 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:28.678 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:28.678 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.16 2024-02-05T05:03:28.678 INFO:tasks.workunit.client.0.smithi148.stderr:58023: instructing pg 1.16 on osd.6 to scrub 2024-02-05T05:03:28.678 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:28.678 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2024-02-05T05:03:28.678 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:28.678 INFO:tasks.workunit.client.0.smithi148.stderr:58023: dumped pgs 2024-02-05T05:03:28.678 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.16 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.678 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.16 2024-02-05T05:03:28.678 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.678 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:28.679 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:28.679 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:28.679 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:28.679 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:28.679 INFO:tasks.workunit.client.0.smithi148.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2024-02-05T05:03:28.679 INFO:tasks.workunit.client.0.smithi148.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-02-05T05:03:28.679 INFO:tasks.workunit.client.0.smithi148.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.679 INFO:tasks.workunit.client.0.smithi148.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-02-05T05:03:28.679 INFO:tasks.workunit.client.0.smithi148.stderr:58023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.679 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:28.679 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:58023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57865 2024-02-05T05:03:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57872 2024-02-05T05:03:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:28.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:28.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:28.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:28.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57882 2024-02-05T05:03:28.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:28.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:28.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:28.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:28.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57885 2024-02-05T05:03:28.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:28.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:28.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:28.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:28.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57894 2024-02-05T05:03:28.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:28.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:28.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:28.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:28.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57901 2024-02-05T05:03:28.683 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1e 2024-02-05T05:03:28.683 INFO:tasks.workunit.client.0.smithi148.stderr:58132: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1e 2024-02-05T05:03:28.683 INFO:tasks.workunit.client.0.smithi148.stderr:58132: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-02-05T05:03:28.684 INFO:tasks.workunit.client.0.smithi148.stderr:58132: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.684 INFO:tasks.workunit.client.0.smithi148.stderr:58132: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.684 INFO:tasks.workunit.client.0.smithi148.stderr:58132: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-02-05T05:03:28.684 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.684 INFO:tasks.workunit.client.0.smithi148.stderr:58132: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:28.684 INFO:tasks.workunit.client.0.smithi148.stderr:58132: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1e query 2024-02-05T05:03:28.684 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:28.684 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:28.684 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:28.684 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1e 2024-02-05T05:03:28.684 INFO:tasks.workunit.client.0.smithi148.stderr:58132: instructing pg 1.1e on osd.3 to scrub 2024-02-05T05:03:28.684 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2024-02-05T05:03:28.684 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:58132: dumped pgs 2024-02-05T05:03:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1e 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1e 2024-02-05T05:03:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:58132: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2024-02-05T05:03:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:58132: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-02-05T05:03:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:58132: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:58132: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-02-05T05:03:28.686 INFO:tasks.workunit.client.0.smithi148.stderr:58132: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:28.686 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:28.686 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:28.686 INFO:tasks.workunit.client.0.smithi148.stderr:58132: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:28.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:29.029 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.9 2024-02-05T05:03:29.030 INFO:tasks.workunit.client.0.smithi148.stderr:57908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.9 2024-02-05T05:03:29.030 INFO:tasks.workunit.client.0.smithi148.stderr:57908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-02-05T05:03:29.030 INFO:tasks.workunit.client.0.smithi148.stderr:57908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.030 INFO:tasks.workunit.client.0.smithi148.stderr:57908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-02-05T05:03:29.030 INFO:tasks.workunit.client.0.smithi148.stderr:57908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.030 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.030 INFO:tasks.workunit.client.0.smithi148.stderr:57908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:29.030 INFO:tasks.workunit.client.0.smithi148.stderr:57908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.9 query 2024-02-05T05:03:29.030 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:29.030 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:29.030 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:29.030 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.9 2024-02-05T05:03:29.030 INFO:tasks.workunit.client.0.smithi148.stderr:57908: instructing pg 1.9 on osd.1 to scrub 2024-02-05T05:03:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2024-02-05T05:03:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:57908: dumped pgs 2024-02-05T05:03:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.9 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.9 2024-02-05T05:03:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:57908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2024-02-05T05:03:29.031 INFO:tasks.workunit.client.0.smithi148.stderr:57908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-02-05T05:03:29.032 INFO:tasks.workunit.client.0.smithi148.stderr:57908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.032 INFO:tasks.workunit.client.0.smithi148.stderr:57908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.032 INFO:tasks.workunit.client.0.smithi148.stderr:57908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-02-05T05:03:29.032 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:03:25.286368+0000 '>' '' 2024-02-05T05:03:29.032 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:29.032 INFO:tasks.workunit.client.0.smithi148.stderr:57908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:29.032 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.14 2024-02-05T05:03:29.032 INFO:tasks.workunit.client.0.smithi148.stderr:58001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.14 2024-02-05T05:03:29.032 INFO:tasks.workunit.client.0.smithi148.stderr:58001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-02-05T05:03:29.032 INFO:tasks.workunit.client.0.smithi148.stderr:58001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.032 INFO:tasks.workunit.client.0.smithi148.stderr:58001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.032 INFO:tasks.workunit.client.0.smithi148.stderr:58001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-02-05T05:03:29.032 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:58001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.14 query 2024-02-05T05:03:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:58001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.14 2024-02-05T05:03:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:58001: instructing pg 1.14 on osd.2 to scrub 2024-02-05T05:03:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2024-02-05T05:03:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:58001: dumped pgs 2024-02-05T05:03:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.14 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.14 2024-02-05T05:03:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:58001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2024-02-05T05:03:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:58001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-02-05T05:03:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:58001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:58001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-02-05T05:03:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:58001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:58001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:29.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:29.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:29.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57905 2024-02-05T05:03:29.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57910 2024-02-05T05:03:29.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57916 2024-02-05T05:03:29.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57926 2024-02-05T05:03:29.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-02-05T05:03:29.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-02-05T05:03:29.037 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 0 + 1 2024-02-05T05:03:29.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=1 2024-02-05T05:03:29.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57933 2024-02-05T05:03:29.055 INFO:tasks.workunit.client.0.smithi148.stderr:et_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.055 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:29.056 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:29.056 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-02-05T05:03:29.056 INFO:tasks.workunit.client.0.smithi148.stderr:58003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-02-05T05:03:29.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-02-05T05:03:29.061 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.13 2024-02-05T05:03:29.061 INFO:tasks.workunit.client.0.smithi148.stderr:57987: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.13 2024-02-05T05:03:29.061 INFO:tasks.workunit.client.0.smithi148.stderr:57987: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-02-05T05:03:29.061 INFO:tasks.workunit.client.0.smithi148.stderr:57987: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.062 INFO:tasks.workunit.client.0.smithi148.stderr:57987: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-02-05T05:03:29.062 INFO:tasks.workunit.client.0.smithi148.stderr:57987: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.062 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.062 INFO:tasks.workunit.client.0.smithi148.stderr:57987: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.13 query 2024-02-05T05:03:29.062 INFO:tasks.workunit.client.0.smithi148.stderr:57987: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:29.062 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:29.062 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:29.062 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:29.062 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.13 2024-02-05T05:03:29.062 INFO:tasks.workunit.client.0.smithi148.stderr:57987: instructing pg 1.13 on osd.4 to scrub 2024-02-05T05:03:29.062 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:29.062 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:29.062 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2024-02-05T05:03:29.062 INFO:tasks.workunit.client.0.smithi148.stderr:57987: dumped pgs 2024-02-05T05:03:29.063 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.13 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.063 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.13 2024-02-05T05:03:29.063 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.063 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:29.063 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:29.063 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:29.063 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:29.063 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:29.063 INFO:tasks.workunit.client.0.smithi148.stderr:57987: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2024-02-05T05:03:29.063 INFO:tasks.workunit.client.0.smithi148.stderr:57987: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-02-05T05:03:29.063 INFO:tasks.workunit.client.0.smithi148.stderr:57987: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.063 INFO:tasks.workunit.client.0.smithi148.stderr:57987: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.063 INFO:tasks.workunit.client.0.smithi148.stderr:57987: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-02-05T05:03:29.064 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:29.064 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:29.064 INFO:tasks.workunit.client.0.smithi148.stderr:57987: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:29.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:29.070 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.10 2024-02-05T05:03:29.070 INFO:tasks.workunit.client.0.smithi148.stderr:57957: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.10 2024-02-05T05:03:29.070 INFO:tasks.workunit.client.0.smithi148.stderr:57957: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-02-05T05:03:29.070 INFO:tasks.workunit.client.0.smithi148.stderr:57957: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.070 INFO:tasks.workunit.client.0.smithi148.stderr:57957: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-02-05T05:03:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:57957: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:57957: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:57957: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.10 query 2024-02-05T05:03:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.10 2024-02-05T05:03:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:57957: instructing pg 1.10 on osd.5 to scrub 2024-02-05T05:03:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2024-02-05T05:03:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:57957: dumped pgs 2024-02-05T05:03:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.10 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.10 2024-02-05T05:03:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:57957: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2024-02-05T05:03:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:57957: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-02-05T05:03:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:57957: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:57957: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-02-05T05:03:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:57957: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.073 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:29.073 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:29.073 INFO:tasks.workunit.client.0.smithi148.stderr:57957: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:29.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:29.076 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1c 2024-02-05T05:03:29.076 INFO:tasks.workunit.client.0.smithi148.stderr:58089: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1c 2024-02-05T05:03:29.076 INFO:tasks.workunit.client.0.smithi148.stderr:58089: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-02-05T05:03:29.076 INFO:tasks.workunit.client.0.smithi148.stderr:58089: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.076 INFO:tasks.workunit.client.0.smithi148.stderr:58089: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-02-05T05:03:29.076 INFO:tasks.workunit.client.0.smithi148.stderr:58089: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.077 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.077 INFO:tasks.workunit.client.0.smithi148.stderr:58089: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1c query 2024-02-05T05:03:29.077 INFO:tasks.workunit.client.0.smithi148.stderr:58089: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:29.077 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:29.077 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:29.077 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:29.077 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1c 2024-02-05T05:03:29.077 INFO:tasks.workunit.client.0.smithi148.stderr:58089: instructing pg 1.1c on osd.0 to scrub 2024-02-05T05:03:29.077 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:29.077 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:29.077 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2024-02-05T05:03:29.077 INFO:tasks.workunit.client.0.smithi148.stderr:58089: dumped pgs 2024-02-05T05:03:29.077 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1c 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1c 2024-02-05T05:03:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:58089: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2024-02-05T05:03:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:58089: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-02-05T05:03:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:58089: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:58089: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-02-05T05:03:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:58089: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:29.079 INFO:tasks.workunit.client.0.smithi148.stderr:58089: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:29.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:29.080 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.18 2024-02-05T05:03:29.080 INFO:tasks.workunit.client.0.smithi148.stderr:58028: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.18 2024-02-05T05:03:29.080 INFO:tasks.workunit.client.0.smithi148.stderr:58028: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-02-05T05:03:29.080 INFO:tasks.workunit.client.0.smithi148.stderr:58028: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.080 INFO:tasks.workunit.client.0.smithi148.stderr:58028: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-02-05T05:03:29.080 INFO:tasks.workunit.client.0.smithi148.stderr:58028: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.18 query 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.18 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: instructing pg 1.18 on osd.5 to scrub 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: dumped pgs 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.18 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.18 2024-02-05T05:03:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:58028: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2024-02-05T05:03:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:58028: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-02-05T05:03:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:58028: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:58028: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-02-05T05:03:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:58028: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.e 2024-02-05T05:03:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:57939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.e 2024-02-05T05:03:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:57939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-02-05T05:03:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:57939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.e query 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.e 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: instructing pg 1.e on osd.1 to scrub 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: dumped pgs 2024-02-05T05:03:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.e 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.084 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.e 2024-02-05T05:03:29.084 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.084 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:29.084 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:29.084 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:29.084 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:29.084 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:29.084 INFO:tasks.workunit.client.0.smithi148.stderr:57939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2024-02-05T05:03:29.084 INFO:tasks.workunit.client.0.smithi148.stderr:57939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-02-05T05:03:29.084 INFO:tasks.workunit.client.0.smithi148.stderr:57939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.084 INFO:tasks.workunit.client.0.smithi148.stderr:57939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.084 INFO:tasks.workunit.client.0.smithi148.stderr:57939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-02-05T05:03:29.085 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scr-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:29.085 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:29.085 INFO:tasks.workunit.client.0.smithi148.stderr:58028: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:29.085 INFO:tasks.workunit.client.0.smithi148.stderr:ub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:29.085 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:29.085 INFO:tasks.workunit.client.0.smithi148.stderr:57939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:29.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:29.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:29.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57940 2024-02-05T05:03:29.731 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1d 2024-02-05T05:03:29.731 INFO:tasks.workunit.client.0.smithi148.stderr:58104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1d 2024-02-05T05:03:29.731 INFO:tasks.workunit.client.0.smithi148.stderr:58104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-02-05T05:03:29.731 INFO:tasks.workunit.client.0.smithi148.stderr:58104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.731 INFO:tasks.workunit.client.0.smithi148.stderr:58104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.731 INFO:tasks.workunit.client.0.smithi148.stderr:58104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-02-05T05:03:29.731 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.731 INFO:tasks.workunit.client.0.smithi148.stderr:58104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:29.731 INFO:tasks.workunit.client.0.smithi148.stderr:58104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1d query 2024-02-05T05:03:29.731 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:29.732 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:29.732 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:29.732 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1d 2024-02-05T05:03:29.732 INFO:tasks.workunit.client.0.smithi148.stderr:58104: instructing pg 1.1d on osd.1 to scrub 2024-02-05T05:03:29.732 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2024-02-05T05:03:29.732 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:29.732 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:29.732 INFO:tasks.workunit.client.0.smithi148.stderr:58104: dumped pgs 2024-02-05T05:03:29.732 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1d 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.732 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1d 2024-02-05T05:03:29.732 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.732 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:29.732 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:29.733 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:29.733 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:29.733 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:29.733 INFO:tasks.workunit.client.0.smithi148.stderr:58104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2024-02-05T05:03:29.733 INFO:tasks.workunit.client.0.smithi148.stderr:58104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-02-05T05:03:29.733 INFO:tasks.workunit.client.0.smithi148.stderr:58104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.733 INFO:tasks.workunit.client.0.smithi148.stderr:58104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.733 INFO:tasks.workunit.client.0.smithi148.stderr:58104: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-02-05T05:03:29.733 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:29.733 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:29.733 INFO:tasks.workunit.client.0.smithi148.stderr:58104: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:29.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:29.734 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1f 2024-02-05T05:03:29.734 INFO:tasks.workunit.client.0.smithi148.stderr:58146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1f 2024-02-05T05:03:29.734 INFO:tasks.workunit.client.0.smithi148.stderr:58146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-02-05T05:03:29.734 INFO:tasks.workunit.client.0.smithi148.stderr:58146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.734 INFO:tasks.workunit.client.0.smithi148.stderr:58146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.734 INFO:tasks.workunit.client.0.smithi148.stderr:58146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-02-05T05:03:29.734 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.734 INFO:tasks.workunit.client.0.smithi148.stderr:58146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:29.734 INFO:tasks.workunit.client.0.smithi148.stderr:58146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1f query 2024-02-05T05:03:29.735 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:29.735 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:29.735 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:29.735 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1f 2024-02-05T05:03:29.735 INFO:tasks.workunit.client.0.smithi148.stderr:58146: instructing pg 1.1f on osd.5 to scrub 2024-02-05T05:03:29.735 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:29.735 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2024-02-05T05:03:29.735 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:29.735 INFO:tasks.workunit.client.0.smithi148.stderr:58146: dumped pgs 2024-02-05T05:03:29.735 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1f 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.735 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1f 2024-02-05T05:03:29.735 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.735 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:29.736 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:29.736 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:29.736 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:29.736 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:29.736 INFO:tasks.workunit.client.0.smithi148.stderr:58146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2024-02-05T05:03:29.736 INFO:tasks.workunit.client.0.smithi148.stderr:58146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-02-05T05:03:29.736 INFO:tasks.workunit.client.0.smithi148.stderr:58146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.736 INFO:tasks.workunit.client.0.smithi148.stderr:58146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-02-05T05:03:29.736 INFO:tasks.workunit.client.0.smithi148.stderr:58146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.736 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:29.736 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:29.736 INFO:tasks.workunit.client.0.smithi148.stderr:58146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:29.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:29.838 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1a 2024-02-05T05:03:29.838 INFO:tasks.workunit.client.0.smithi148.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1a 2024-02-05T05:03:29.838 INFO:tasks.workunit.client.0.smithi148.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-02-05T05:03:29.838 INFO:tasks.workunit.client.0.smithi148.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.838 INFO:tasks.workunit.client.0.smithi148.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-02-05T05:03:29.838 INFO:tasks.workunit.client.0.smithi148.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.838 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.838 INFO:tasks.workunit.client.0.smithi148.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:29.839 INFO:tasks.workunit.client.0.smithi148.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1a query 2024-02-05T05:03:29.839 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:29.839 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:29.839 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:29.839 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1a 2024-02-05T05:03:29.839 INFO:tasks.workunit.client.0.smithi148.stderr:58061: instructing pg 1.1a on osd.1 to scrub 2024-02-05T05:03:29.839 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:29.839 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:29.839 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2024-02-05T05:03:29.839 INFO:tasks.workunit.client.0.smithi148.stderr:58061: dumped pgs 2024-02-05T05:03:29.839 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1a 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.839 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1a 2024-02-05T05:03:29.839 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.839 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:29.840 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:29.840 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:29.840 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:29.840 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:29.840 INFO:tasks.workunit.client.0.smithi148.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2024-02-05T05:03:29.840 INFO:tasks.workunit.client.0.smithi148.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-02-05T05:03:29.840 INFO:tasks.workunit.client.0.smithi148.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.840 INFO:tasks.workunit.client.0.smithi148.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.840 INFO:tasks.workunit.client.0.smithi148.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-02-05T05:03:29.840 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:03:26.421053+0000 '>' '' 2024-02-05T05:03:29.840 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:29.840 INFO:tasks.workunit.client.0.smithi148.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:29.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:29.851 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.f 2024-02-05T05:03:29.851 INFO:tasks.workunit.client.0.smithi148.stderr:57949: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.f 2024-02-05T05:03:29.851 INFO:tasks.workunit.client.0.smithi148.stderr:57949: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-02-05T05:03:29.851 INFO:tasks.workunit.client.0.smithi148.stderr:57949: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.851 INFO:tasks.workunit.client.0.smithi148.stderr:57949: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.851 INFO:tasks.workunit.client.0.smithi148.stderr:57949: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-02-05T05:03:29.851 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.851 INFO:tasks.workunit.client.0.smithi148.stderr:57949: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.f query 2024-02-05T05:03:29.851 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T05:03:29.851 INFO:tasks.workunit.client.0.smithi148.stderr:57949: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T05:03:29.851 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T05:03:29.852 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T05:03:29.852 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.f 2024-02-05T05:03:29.852 INFO:tasks.workunit.client.0.smithi148.stderr:57949: instructing pg 1.f on osd.6 to scrub 2024-02-05T05:03:29.852 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2024-02-05T05:03:29.852 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T05:03:29.852 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T05:03:29.852 INFO:tasks.workunit.client.0.smithi148.stderr:57949: dumped pgs 2024-02-05T05:03:29.852 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.f 2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.852 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.f 2024-02-05T05:03:29.852 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T05:02:02.024198+0000 2024-02-05T05:03:29.852 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T05:03:29.852 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T05:03:29.852 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T05:03:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T05:03:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T05:03:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:57949: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2024-02-05T05:03:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:57949: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-02-05T05:03:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:57949: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:03:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:57949: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-02-05T05:03:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:57949: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:03:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T05:02:02.024198+0000 '>' '' 2024-02-05T05:03:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T05:03:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:57949: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T05:03:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:03:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57951 2024-02-05T05:03:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57953 2024-02-05T05:03:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57962 2024-02-05T05:03:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-02-05T05:03:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-02-05T05:03:29.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 1 + 1 2024-02-05T05:03:29.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=2 2024-02-05T05:03:29.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57975 2024-02-05T05:03:29.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57988 2024-02-05T05:03:29.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 57998 2024-02-05T05:03:29.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-02-05T05:03:29.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-02-05T05:03:29.856 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 2 + 1 2024-02-05T05:03:29.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=3 2024-02-05T05:03:29.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58005 2024-02-05T05:03:29.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58014 2024-02-05T05:03:29.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58024 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58033 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58043 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58063 2024-02-05T05:03:29.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58078 2024-02-05T05:03:29.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58097 2024-02-05T05:03:29.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58121 2024-02-05T05:03:29.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T05:03:29.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58133 2024-02-05T05:03:29.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T05:03:29.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T05:03:29.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T05:03:29.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: eval 'pids='\'''\''' 2024-02-05T05:03:29.859 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: pids= 2024-02-05T05:03:29.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:229: wait_background_check: return 0 2024-02-05T05:03:29.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:307: TEST_recovery_scrub_2: return_code=0 2024-02-05T05:03:29.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:308: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2024-02-05T05:03:29.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: ERRORS=0 2024-02-05T05:03:29.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:311: TEST_recovery_scrub_2: test 3 -eq 0 2024-02-05T05:03:29.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2024-02-05T05:03:29.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2024-02-05T05:03:29.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/osd.5.pid 2024-02-05T05:03:29.861 INFO:tasks.workunit.client.0.smithi148.stderr:td/osd-recovery-scrub/osd.2.pid 2024-02-05T05:03:29.861 INFO:tasks.workunit.client.0.smithi148.stderr:td/osd-recovery-scrub/osd.3.pid 2024-02-05T05:03:29.861 INFO:tasks.workunit.client.0.smithi148.stderr:td/osd-recovery-scrub/mgr.x.pid 2024-02-05T05:03:29.861 INFO:tasks.workunit.client.0.smithi148.stderr:td/osd-recovery-scrub/osd.7.pid 2024-02-05T05:03:29.862 INFO:tasks.workunit.client.0.smithi148.stderr:td/osd-recovery-scrub/osd.4.pid 2024-02-05T05:03:29.862 INFO:tasks.workunit.client.0.smithi148.stderr:td/osd-recovery-scrub/mon.a.pid 2024-02-05T05:03:29.862 INFO:tasks.workunit.client.0.smithi148.stderr:td/osd-recovery-scrub/osd.1.pid 2024-02-05T05:03:29.862 INFO:tasks.workunit.client.0.smithi148.stderr:td/osd-recovery-scrub/osd.0.pid 2024-02-05T05:03:29.862 INFO:tasks.workunit.client.0.smithi148.stderr:td/osd-recovery-scrub/osd.6.pid' 2024-02-05T05:03:29.862 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.2.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.6.pid 2024-02-05T05:03:29.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: pid='52899 2024-02-05T05:03:29.862 INFO:tasks.workunit.client.0.smithi148.stderr:48771 2024-02-05T05:03:29.862 INFO:tasks.workunit.client.0.smithi148.stderr:50147 2024-02-05T05:03:29.862 INFO:tasks.workunit.client.0.smithi148.stderr:45076 2024-02-05T05:03:29.862 INFO:tasks.workunit.client.0.smithi148.stderr:55653 2024-02-05T05:03:29.862 INFO:tasks.workunit.client.0.smithi148.stderr:51523 2024-02-05T05:03:29.862 INFO:tasks.workunit.client.0.smithi148.stderr:44992 2024-02-05T05:03:29.862 INFO:tasks.workunit.client.0.smithi148.stderr:47395 2024-02-05T05:03:29.863 INFO:tasks.workunit.client.0.smithi148.stderr:46019 2024-02-05T05:03:29.863 INFO:tasks.workunit.client.0.smithi148.stderr:54277' 2024-02-05T05:03:29.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: kill -0 52899 48771 50147 45076 55653 51523 44992 47395 46019 54277 2024-02-05T05:03:29.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:327: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2024-02-05T05:03:29.863 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:03:29.863 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:03:29.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:03:29.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:03:29.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:03:31.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:03:31.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: declare -a err_strings 2024-02-05T05:03:31.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:330: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2024-02-05T05:03:31.168 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: expr 8 - 1 2024-02-05T05:03:31.169 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: seq 0 7 2024-02-05T05:03:31.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2024-02-05T05:03:31.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2024-02-05T05:03:31.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2024-02-05T05:03:31.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2024-02-05T05:03:31.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2024-02-05T05:03:31.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2024-02-05T05:03:31.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2024-02-05T05:03:31.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2024-02-05T05:03:31.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:336: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2024-02-05T05:03:31.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:338: TEST_recovery_scrub_2: found=false 2024-02-05T05:03:31.210 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: expr 8 - 1 2024-02-05T05:03:31.211 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: seq 0 7 2024-02-05T05:03:31.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2024-02-05T05:03:31.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2024-02-05T05:03:31.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2024-02-05T05:03:31.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2024-02-05T05:03:31.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2024-02-05T05:03:31.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2024-02-05T05:03:31.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2024-02-05T05:03:31.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:03:31.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2024-02-05T05:03:31.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:346: TEST_recovery_scrub_2: '[' false = true ']' 2024-02-05T05:03:31.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:352: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2024-02-05T05:03:31.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-02-05T05:03:31.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:03:31.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-02-05T05:03:31.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:03:31.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:03:31.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:03:31.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:03:31.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:03:31.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:03:31.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:03:31.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:03:31.256 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:03:31.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:03:31.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:03:31.258 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:03:31.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:03:31.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:03:31.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:03:31.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:03:31.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:03:31.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:03:31.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-02-05T05:03:31.285 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:03:31.285 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:03:31.285 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:03:31.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36362 2024-02-05T05:03:31.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:03:31.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:03:31.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:354: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2024-02-05T05:03:31.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:360: TEST_recovery_scrub_2: echo 'TEST PASSED' 2024-02-05T05:03:31.287 INFO:tasks.workunit.client.0.smithi148.stdout:TEST PASSED 2024-02-05T05:03:31.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:361: TEST_recovery_scrub_2: return 0 2024-02-05T05:03:31.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-05T05:03:31.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-recovery-scrub 0 2024-02-05T05:03:31.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-02-05T05:03:31.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-05T05:03:31.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-02-05T05:03:31.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:03:31.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:03:31.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:03:31.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:03:31.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:03:31.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:03:31.290 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:03:31.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:03:31.291 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:03:31.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:03:31.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:03:31.292 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:03:31.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:03:31.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:03:31.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:03:31.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:03:31.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:03:31.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-05T05:03:31.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-02-05T05:03:31.296 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:03:31.296 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:03:31.296 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36362 2024-02-05T05:03:31.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36362 2024-02-05T05:03:31.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:03:31.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:03:31.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-05T05:03:31.298 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-05T05:03:31.298 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-05T05:03:31.352 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-02-05T05:03:31.353 DEBUG:teuthology.orchestra.run.smithi148:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-02-05T05:03:31.404 INFO:tasks.workunit.client.0.smithi148.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-05T05:03:31.408 INFO:tasks.workunit.client.0.smithi148.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-05T05:03:31.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-05T05:03:31.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-05T05:03:31.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-05T05:03:31.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-05T05:03:31.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-05T05:03:31.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-05T05:03:31.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-05T05:03:31.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-05T05:03:31.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-dump 2024-02-05T05:03:31.409 INFO:tasks.workunit.client.0.smithi148.stdout:This test is disabled 2024-02-05T05:03:31.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-02-05T05:03:31.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-02-05T05:03:31.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-05T05:03:31.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-dump 0 2024-02-05T05:03:31.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-02-05T05:03:31.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-05T05:03:31.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-02-05T05:03:31.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:03:31.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:03:31.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:03:31.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:03:31.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:03:31.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:03:31.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:03:31.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:03:31.413 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:03:31.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:03:31.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:03:31.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:03:31.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:03:31.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:03:31.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:03:31.415 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:03:31.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:03:31.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-05T05:03:31.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-02-05T05:03:31.418 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:03:31.418 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:03:31.418 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61331 2024-02-05T05:03:31.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61331 2024-02-05T05:03:31.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:03:31.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:03:31.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-05T05:03:31.420 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-05T05:03:31.420 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-05T05:03:31.472 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-02-05T05:03:31.472 DEBUG:teuthology.orchestra.run.smithi148:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-02-05T05:03:31.524 INFO:tasks.workunit.client.0.smithi148.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-02-05T05:03:31.524 INFO:tasks.workunit.client.0.smithi148.stderr:++ TIMEOUT=300 2024-02-05T05:03:31.524 INFO:tasks.workunit.client.0.smithi148.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-02-05T05:03:31.524 INFO:tasks.workunit.client.0.smithi148.stderr:++ MAX_TIMEOUT=15 2024-02-05T05:03:31.524 INFO:tasks.workunit.client.0.smithi148.stderr:++ PG_NUM=4 2024-02-05T05:03:31.524 INFO:tasks.workunit.client.0.smithi148.stderr:++ TMPDIR=/tmp 2024-02-05T05:03:31.524 INFO:tasks.workunit.client.0.smithi148.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-02-05T05:03:31.525 INFO:tasks.workunit.client.0.smithi148.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-02-05T05:03:31.525 INFO:tasks.workunit.client.0.smithi148.stderr:++ type xmlstarlet 2024-02-05T05:03:31.525 INFO:tasks.workunit.client.0.smithi148.stderr:++ XMLSTARLET=xmlstarlet 2024-02-05T05:03:31.525 INFO:tasks.workunit.client.0.smithi148.stderr:+++ uname 2024-02-05T05:03:31.526 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' Linux = FreeBSD ']' 2024-02-05T05:03:31.526 INFO:tasks.workunit.client.0.smithi148.stderr:++ SED=sed 2024-02-05T05:03:31.526 INFO:tasks.workunit.client.0.smithi148.stderr:++ AWK=awk 2024-02-05T05:03:31.526 INFO:tasks.workunit.client.0.smithi148.stderr:+++ stty -a 2024-02-05T05:03:31.526 INFO:tasks.workunit.client.0.smithi148.stderr:+++ head -1 2024-02-05T05:03:31.526 INFO:tasks.workunit.client.0.smithi148.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-02-05T05:03:31.526 INFO:tasks.workunit.client.0.smithi148.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-05T05:03:31.527 INFO:tasks.workunit.client.0.smithi148.stderr:++ termwidth= 2024-02-05T05:03:31.527 INFO:tasks.workunit.client.0.smithi148.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-02-05T05:03:31.528 INFO:tasks.workunit.client.0.smithi148.stderr:++ DIFFCOLOPTS='-y ' 2024-02-05T05:03:31.528 INFO:tasks.workunit.client.0.smithi148.stderr:++ KERNCORE=kernel.core_pattern 2024-02-05T05:03:31.528 INFO:tasks.workunit.client.0.smithi148.stderr:++ EXTRA_OPTS= 2024-02-05T05:03:31.532 INFO:tasks.workunit.client.0.smithi148.stderr:++ test '' = TESTS 2024-02-05T05:03:31.532 INFO:tasks.workunit.client.0.smithi148.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-02-05T05:03:31.533 INFO:tasks.workunit.client.0.smithi148.stderr:++ uname 2024-02-05T05:03:31.534 INFO:tasks.workunit.client.0.smithi148.stderr:+ '[' Linux = FreeBSD ']' 2024-02-05T05:03:31.534 INFO:tasks.workunit.client.0.smithi148.stderr:+ use_ec_overwrite=true 2024-02-05T05:03:31.535 INFO:tasks.workunit.client.0.smithi148.stderr:+ getjson=no 2024-02-05T05:03:31.535 INFO:tasks.workunit.client.0.smithi148.stderr:+ jqfilter='def walk(f): 2024-02-05T05:03:31.535 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T05:03:31.535 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T05:03:31.535 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T05:03:31.535 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:03:31.535 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:03:31.535 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T05:03:31.535 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T05:03:31.535 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:03:31.535 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:03:31.535 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:03:31.535 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:03:31.536 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T05:03:31.536 INFO:tasks.workunit.client.0.smithi148.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:03:31.542 INFO:tasks.workunit.client.0.smithi148.stderr:+ main osd-scrub-repair 2024-02-05T05:03:31.542 INFO:tasks.workunit.client.0.smithi148.stderr:+ local dir=td/osd-scrub-repair 2024-02-05T05:03:31.542 INFO:tasks.workunit.client.0.smithi148.stderr:+ shift 2024-02-05T05:03:31.542 INFO:tasks.workunit.client.0.smithi148.stderr:+ shopt -s -o xtrace 2024-02-05T05:03:31.542 INFO:tasks.workunit.client.0.smithi148.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-05T05:03:31.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-05T05:03:31.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-05T05:03:31.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-05T05:03:31.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-05T05:03:31.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-05T05:03:31.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-05T05:03:31.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-05T05:03:31.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-05T05:03:31.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-repair 2024-02-05T05:03:31.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-02-05T05:03:31.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-02-05T05:03:31.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-02-05T05:03:31.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-02-05T05:03:31.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-02-05T05:03:31.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-02-05T05:03:31.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none ' 2024-02-05T05:03:31.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-02-05T05:03:31.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-02-05T05:03:31.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-02-05T05:03:31.545 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-02-05T05:03:31.545 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-05T05:03:31.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-02-05T05:03:31.549 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_auto_repair_bluestore_basic 2024-02-05T05:03:31.549 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_auto_repair_bluestore_failed 2024-02-05T05:03:31.550 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-02-05T05:03:31.550 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_auto_repair_bluestore_scrub 2024-02-05T05:03:31.550 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_auto_repair_bluestore_tag 2024-02-05T05:03:31.550 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_auto_repair_erasure_coded_appends 2024-02-05T05:03:31.550 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-02-05T05:03:31.550 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-02-05T05:03:31.550 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-02-05T05:03:31.550 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_corrupt_and_repair_lrc_appends 2024-02-05T05:03:31.550 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-02-05T05:03:31.550 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_corrupt_and_repair_replicated 2024-02-05T05:03:31.550 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_corrupt_scrub_erasure_appends 2024-02-05T05:03:31.550 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-02-05T05:03:31.550 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_corrupt_scrub_replicated 2024-02-05T05:03:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_corrupt_snapset_scrub_rep 2024-02-05T05:03:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_list_missing_erasure_coded_appends 2024-02-05T05:03:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_list_missing_erasure_coded_overwrites 2024-02-05T05:03:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_periodic_scrub_replicated 2024-02-05T05:03:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_repair_stats 2024-02-05T05:03:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_repair_stats_ec 2024-02-05T05:03:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_request_scrub_priority 2024-02-05T05:03:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_scrub_warning 2024-02-05T05:03:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_skip_non_repair_during_recovery 2024-02-05T05:03:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_unfound_erasure_coded_appends 2024-02-05T05:03:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_unfound_erasure_coded_overwrites' 2024-02-05T05:03:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:03:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:03:31.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:03:31.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:03:31.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:03:31.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:03:31.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:03:31.552 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:03:31.552 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:03:31.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:03:31.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:03:31.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:03:31.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:03:31.553 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:03:31.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:03:31.553 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:03:31.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:03:31.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:03:31.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:03:31.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:03:31.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:03:31.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:03:31.556 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:03:31.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:03:31.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:03:31.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:03:31.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:03:31.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:03:31.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:03:31.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:03:31.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:03:31.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:03:31.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:03:31.562 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:03:31.562 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:03:31.562 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:03:31.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T05:03:31.565 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:03:31.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-05T05:03:31.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-05T05:03:31.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:03:31.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:03:31.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-02-05T05:03:31.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-02-05T05:03:31.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-02-05T05:03:31.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-02-05T05:03:31.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:03:31.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:03:31.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:03:31.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:03:31.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:03:31.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-02-05T05:03:31.950 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:03:31.950 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:03:31.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:03:31.952 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:03:31.952 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:03:31.952 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:03:31.952 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:03:31.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T05:03:32.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:03:32.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:03:32.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:03:32.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:03:32.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:03:32.162 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:03:32.162 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:03:32.162 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:03:32.162 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:03:32.163 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:03:32.163 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:03:32.163 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:03:32.163 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:03:32.163 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:03:32.163 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T05:03:32.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:03:32.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:03:32.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:03:32.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:03:32.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:03:32.280 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:03:32.280 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:03:32.280 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:03:32.280 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:03:32.280 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:03:32.280 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:03:32.280 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:03:32.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:03:32.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T05:03:32.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-02-05T05:03:32.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:03:32.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:03:32.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:03:32.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:03:32.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:03:32.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:03:33.253 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:03:33.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:03:33.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:03:33.254 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:03:33.254 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:03:33.255 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:03:33.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:03:33.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:03:33.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:03:33.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T05:03:33.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:03:33.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:03:33.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:03:33.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:03:33.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:03:33.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:03:33.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:03:33.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:03:33.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:03:33.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:03:33.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:03:33.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:03:33.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:03:33.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:03:33.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:03:33.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:03:33.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:03:33.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:03:33.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-02-05T05:03:33.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:03:33.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:03:33.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=17f59e03-d508-4533-ab12-8532fbc86636 2024-02-05T05:03:33.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 17f59e03-d508-4533-ab12-8532fbc86636' 2024-02-05T05:03:33.295 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 17f59e03-d508-4533-ab12-8532fbc86636 2024-02-05T05:03:33.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:03:33.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlbMBlFWXBEhAAPNWqZHfpfOu4IJqb0hVAGQ== 2024-02-05T05:03:33.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlbMBlFWXBEhAAPNWqZHfpfOu4IJqb0hVAGQ=="}' 2024-02-05T05:03:33.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 17f59e03-d508-4533-ab12-8532fbc86636 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:03:33.549 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:03:33.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:03:33.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQAlbMBlFWXBEhAAPNWqZHfpfOu4IJqb0hVAGQ== --osd-uuid 17f59e03-d508-4533-ab12-8532fbc86636 2024-02-05T05:03:33.599 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:03:33.602+0000 7f83809d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:33.602 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:03:33.606+0000 7f83809d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:33.604 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:03:33.606+0000 7f83809d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:33.604 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:03:33.606+0000 7f83809d2640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:03:33.604 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:03:33.606+0000 7f83809d2640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:03:33.604 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:03:33.606+0000 7f83809d2640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:03:33.695 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:03:33.698+0000 7f83809d2640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:03:39.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:03:39.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:03:39.557 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:03:39.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:03:39.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:03:39.973 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:03:39.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:03:39.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:03:39.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:03:39.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T05:03:39.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:03:40.008 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:03:40.010+0000 7f0519477640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:40.015 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:03:40.018+0000 7f0519477640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:40.016 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:03:40.018+0000 7f0519477640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:40.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:03:40.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:03:40.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:03:40.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:03:40.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:03:40.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:40.266 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:03:40.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:03:40.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:40.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:03:40.564 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:03:40.566+0000 7f0519477640 -1 Falling back to public interface 2024-02-05T05:03:40.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:41.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:41.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:41.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:03:41.582 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:03:41.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:41.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:03:41.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:42.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:42.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:42.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:03:42.883 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:03:42.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:42.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:03:43.067 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:03:43.070+0000 7f0519477640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:03:43.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:44.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:44.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:44.194 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:03:44.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:03:44.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:44.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:03:44.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:45.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:45.493 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:03:45.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:45.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:03:45.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:45.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:03:45.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:46.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:46.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:46.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:03:46.835 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:03:46.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:46.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:03:47.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:48.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:48.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:48.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:03:48.153 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:03:48.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:48.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:03:48.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:49.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:49.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:49.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:03:49.454 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:03:49.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:49.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:03:49.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:50.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:50.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:50.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:03:50.774 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:03:50.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:50.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:03:51.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:52.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:52.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:52.094 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:03:52.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:03:52.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:52.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:03:52.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:53.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:53.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:53.408 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:03:53.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:03:53.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:53.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:03:53.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:54.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:54.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:54.782 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:03:54.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:03:54.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:54.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:03:55.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:56.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:56.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:56.232 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:03:56.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:03:56.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:56.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:03:56.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:57.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:57.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:57.772 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:03:57.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:03:57.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:57.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:03:58.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:59.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:59.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:59.547 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:03:59.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:03:59.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:59.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:04:00.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:01.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:01.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:01.037 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:04:01.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:04:01.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:01.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:04:01.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:02.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:02.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:02.655 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:04:02.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:04:02.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:02.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:04:03.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:04.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:04.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:04.488 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:04:04.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:04:04.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:04.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:04:05.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:06.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:06.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:06.146 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:04:06.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:04:06.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:06.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:04:06.492 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/505308788,v1:127.0.0.1:6803/505308788] [v2:127.0.0.1:6804/505308788,v1:127.0.0.1:6805/505308788] exists,up 17f59e03-d508-4533-ab12-8532fbc86636 2024-02-05T05:04:06.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:04:06.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:04:06.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:04:06.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T05:04:06.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:04:06.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:04:06.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:04:06.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:04:06.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:04:06.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:04:06.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:04:06.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:04:06.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:04:06.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:04:06.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:04:06.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:04:06.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:04:06.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:04:06.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:04:06.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:04:06.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:04:06.497 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:04:06.497 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:04:06.497 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:04:06.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:04:06.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:04:06.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:04:06.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:04:06.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:04:06.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:04:06.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:04:06.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:04:06.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:04:06.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:04:06.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:04:06.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:04:06.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-02-05T05:04:06.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:04:06.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:04:06.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d615516f-a8df-4af9-97d1-47f28b108d25 2024-02-05T05:04:06.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d615516f-a8df-4af9-97d1-47f28b108d25' 2024-02-05T05:04:06.499 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 d615516f-a8df-4af9-97d1-47f28b108d25 2024-02-05T05:04:06.499 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:04:06.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGbMBlDbHsHhAAwM9S31mtBAWKmiu7cuQCiw== 2024-02-05T05:04:06.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGbMBlDbHsHhAAwM9S31mtBAWKmiu7cuQCiw=="}' 2024-02-05T05:04:06.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d615516f-a8df-4af9-97d1-47f28b108d25 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:04:06.874 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:04:06.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:04:06.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBGbMBlDbHsHhAAwM9S31mtBAWKmiu7cuQCiw== --osd-uuid d615516f-a8df-4af9-97d1-47f28b108d25 2024-02-05T05:04:06.936 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:06.938+0000 7f93525a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:06.950 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:06.954+0000 7f93525a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:06.951 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:06.954+0000 7f93525a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:06.952 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:06.954+0000 7f93525a1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:04:06.952 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:06.954+0000 7f93525a1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:04:06.952 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:06.954+0000 7f93525a1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:04:07.025 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:07.026+0000 7f93525a1640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:04:13.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:04:13.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:04:13.254 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:04:13.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:04:13.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:04:13.633 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:04:13.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:04:13.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T05:04:13.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:04:13.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:04:13.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:04:13.667 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:13.670+0000 7fc67d2f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:13.691 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:13.694+0000 7fc67d2f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:13.692 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:13.694+0000 7fc67d2f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:13.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:04:13.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:04:13.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:04:13.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:04:13.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:04:13.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:13.982 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:04:13.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:04:13.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:13.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:14.248 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:14.250+0000 7fc67d2f4640 -1 Falling back to public interface 2024-02-05T05:04:14.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:15.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:15.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:15.294 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:04:15.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:04:15.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:15.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:15.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:16.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:16.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:16.600 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:04:16.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:04:16.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:16.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:16.840 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:16.842+0000 7fc67d2f4640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:04:16.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:17.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:17.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:17.940 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:04:17.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:04:17.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:17.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:18.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:19.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:19.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:19.251 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:04:19.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:04:19.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:19.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:19.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:20.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:20.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:20.667 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:04:20.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:04:20.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:20.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:20.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:21.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:21.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:21.976 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:04:21.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:04:21.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:21.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:22.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:23.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:23.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:23.296 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:04:23.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:04:23.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:23.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:23.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:24.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:24.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:24.689 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:04:24.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:04:24.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:24.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:25.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:26.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:26.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:26.007 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:04:26.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:04:26.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:26.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:26.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:27.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:27.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:27.321 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:04:27.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:04:27.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:27.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:27.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:28.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:28.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:28.697 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:04:28.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:04:28.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:28.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:29.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:30.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:30.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:30.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:04:30.018 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:04:30.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:30.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:30.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:31.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:31.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:31.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:04:31.886 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:04:31.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:31.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:32.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:33.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:33.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:33.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:04:33.597 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:04:33.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:33.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:34.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:35.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:35.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:35.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:04:35.143 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:04:35.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:35.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:35.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:36.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:36.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:36.815 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:04:36.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:04:36.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:36.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:37.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:38.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:38.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:38.501 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:04:38.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:04:38.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:38.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:39.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:40.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:40.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:40.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:04:40.191 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:04:40.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:40.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:40.504 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2353972227,v1:127.0.0.1:6811/2353972227] [v2:127.0.0.1:6812/2353972227,v1:127.0.0.1:6813/2353972227] exists,up d615516f-a8df-4af9-97d1-47f28b108d25 2024-02-05T05:04:40.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:04:40.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:04:40.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:04:40.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-02-05T05:04:40.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:04:40.801 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T05:04:40.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:04:40.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:04:41.238 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T05:04:41.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:04:42.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:04:44.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-02-05T05:04:44.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:04:44.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:04:44.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:04:44.075 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:04:44.076 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:04:44.076 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:04:44.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:04:44.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:04:44.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:04:44.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:04:44.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:04:44.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:04:44.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:04:44.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:04:44.201 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:04:44.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:04:44.515 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T05:04:44.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:04:44.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:44.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:04:44.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-02-05T05:04:44.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-02-05T05:04:44.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-02-05T05:04:44.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:44.663 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:04:44.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-02-05T05:04:44.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-02-05T05:04:44.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672965' 2024-02-05T05:04:44.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:44.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-02-05T05:04:44.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:44.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:04:44.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-02-05T05:04:44.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:44.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-02-05T05:04:44.806 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836492 2024-02-05T05:04:44.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-02-05T05:04:44.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:04:45.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-02-05T05:04:45.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:04:46.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:04:46.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:04:46.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-02-05T05:04:46.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:46.404 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-02-05T05:04:46.404 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:46.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:04:46.406 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-02-05T05:04:46.406 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:46.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-02-05T05:04:46.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-02-05T05:04:46.407 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672965 2024-02-05T05:04:46.408 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:04:46.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672965 2024-02-05T05:04:46.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:04:46.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:04:46.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:04:47.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:04:47.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:04:47.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:04:47.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:04:47.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:04:47.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:04:47.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:04:47.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:04:47.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:04:47.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:04:47.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:04:47.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:04:47.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:04:47.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:04:47.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:04:47.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-02-05T05:04:47.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:04:47.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-02-05T05:04:47.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T05:04:47.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:04:47.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:04:47.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:04:48.176 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T05:04:48.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:04:48.889 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T05:04:48.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:04:48.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:04:48.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:04:49.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-02-05T05:04:49.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-02-05T05:04:49.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:04:49.171 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-02-05T05:04:49.171 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-02-05T05:04:49.171 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:04:49.172 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-02-05T05:04:49.172 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:04:49.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:04:49.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-02-05T05:04:49.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:04:49.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-02-05T05:04:49.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:04:49.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-02-05T05:04:49.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-02-05T05:04:49.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:04:49.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:04:49.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:04:49.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:04:49.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:04:49.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:04:49.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:04:49.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:04:49.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:04:49.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:04:49.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:04:49.809 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:04:49.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:04:49.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:04:49.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:04:49.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:04:50.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:04:50.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:04:50.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:04:50.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:04:50.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:04:50.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:04:50.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:04:50.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-02-05T05:04:51.418 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-02-05T05:04:51.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:04:51.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:04:51.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:04:51.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:04:51.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:04:51.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:04:51.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:04:51.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:04:51.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:04:51.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:04:51.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:04:51.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:04:51.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:04:51.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:04:51.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:04:51.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:04:51.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:04:51.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:04:51.959 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:04:51.959 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:04:51.959 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:04:51.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:04:51.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:04:51.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:04:51.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:04:51.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:04:51.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:04:51.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:04:51.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:04:51.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:04:51.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:04:51.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:04:51.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:04:51.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:04:51.962 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:04:51.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:04:51.962 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:04:51.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:04:51.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:04:51.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:04:51.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:04:51.994 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:51.997+0000 7f670058a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:51.997 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:52.001+0000 7f670058a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:51.998 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:52.001+0000 7f670058a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:52.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:04:52.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:04:52.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:04:52.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:04:52.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:04:52.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:52.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:04:52.282 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:04:52.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:04:52.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:52.531 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:52.533+0000 7f670058a640 -1 Falling back to public interface 2024-02-05T05:04:52.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:53.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:53.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:53.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:04:53.586 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:04:53.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:53.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:04:53.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:54.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:54.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:54.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:04:54.910 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:04:54.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:54.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:04:55.058 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:04:55.061+0000 7f670058a640 -1 osd.0 22 log_to_monitors true 2024-02-05T05:04:55.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:56.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:56.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:56.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:04:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:04:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:04:56.627 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/3584520892,v1:127.0.0.1:6803/3584520892] [v2:127.0.0.1:6804/3584520892,v1:127.0.0.1:6805/3584520892] exists,up 17f59e03-d508-4533-ab12-8532fbc86636 2024-02-05T05:04:56.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:04:56.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:04:56.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:04:56.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:04:56.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:04:56.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:04:56.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:04:56.627 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:04:56.628 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:04:56.628 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:04:56.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:04:56.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:04:56.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:04:56.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:04:56.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:04:56.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:04:56.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:04:56.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:04:56.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:04:57.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:04:57.038 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T05:04:57.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:04:57.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:57.039 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:04:57.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-02-05T05:04:57.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-02-05T05:04:57.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-02-05T05:04:57.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:57.196 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:04:57.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-02-05T05:04:57.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-02-05T05:04:57.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672969' 2024-02-05T05:04:57.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:57.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-02-05T05:04:57.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:57.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:04:57.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-02-05T05:04:57.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:57.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-02-05T05:04:57.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-02-05T05:04:57.343 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 111669149698 2024-02-05T05:04:57.343 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:04:57.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-02-05T05:04:57.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:04:58.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:04:58.646 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:04:58.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-02-05T05:04:58.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:58.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-02-05T05:04:58.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:58.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:04:58.947 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-02-05T05:04:58.948 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:58.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-02-05T05:04:58.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-02-05T05:04:58.949 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672969 2024-02-05T05:04:58.949 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:04:59.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672969 2024-02-05T05:04:59.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:04:59.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:04:59.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:04:59.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:04:59.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:04:59.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:04:59.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:04:59.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:04:59.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:04:59.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:04:59.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:04:59.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:04:59.923 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:04:59.923 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:04:59.923 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:00.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:05:00.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:05:00.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:05:00.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-02-05T05:05:00.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-02-05T05:05:00.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:05:00.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-02-05T05:05:00.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:05:00.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-02-05T05:05:00.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-02-05T05:05:00.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-02-05T05:05:00.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-02-05T05:05:00.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:05:00.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:05:00.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:05:00.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:05:00.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:04:41.181911+0000 2024-02-05T05:05:00.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-02-05T05:05:01.211 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 1.3 on osd.1 to repair 2024-02-05T05:05:01.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-02-05T05:04:41.181911+0000 2024-02-05T05:05:01.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-02-05T05:05:01.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:04:41.181911+0000 2024-02-05T05:05:01.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:05:01.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:05:01.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:05:01.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:05:01.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:05:01.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:05:01.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:05:01.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:05:01.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:04:41.181911+0000 '>' 2024-02-05T05:04:41.181911+0000 2024-02-05T05:05:01.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:05:02.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:05:02.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:05:02.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:05:02.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:05:02.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:05:02.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:05:02.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:05:02.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:04:41.181911+0000 '>' 2024-02-05T05:04:41.181911+0000 2024-02-05T05:05:02.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:05:03.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:05:03.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:05:03.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:05:03.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:05:03.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:05:03.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:05:03.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:05:04.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:04:41.181911+0000 '>' 2024-02-05T05:04:41.181911+0000 2024-02-05T05:05:04.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:05:05.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:05:05.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:05:05.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:05:05.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:05:05.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:05:05.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:05:05.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:05:05.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:04:41.181911+0000 '>' 2024-02-05T05:04:41.181911+0000 2024-02-05T05:05:05.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:05:06.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:05:06.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:05:06.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:05:06.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:05:06.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:05:06.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:05:06.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:05:06.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:05:01.838123+0000 '>' 2024-02-05T05:04:41.181911+0000 2024-02-05T05:05:06.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:05:06.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:05:06.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:05:06.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:05:06.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:05:06.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:05:06.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:05:06.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:05:06.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:05:06.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:05:06.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:05:06.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:05:06.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:05:06.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:05:06.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:05:06.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:05:06.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:05:07.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:05:07.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:05:07.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:05:07.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:05:07.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:05:07.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:05:07.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:05:07.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:05:08.258 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-02-05T05:05:08.259 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:05:08.259 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:05:08.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:05:08.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:05:08.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:05:08.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:05:08.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:05:08.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:05:08.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:05:08.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:05:08.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:05:08.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:05:08.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:05:08.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:05:08.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:05:08.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:05:08.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:05:08.799 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:05:08.799 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:05:08.799 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:05:08.799 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:05:08.799 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:05:08.799 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:05:08.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:05:08.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:05:08.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:05:08.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:05:08.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:05:08.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:05:08.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:05:08.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:05:08.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:05:08.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:05:08.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:05:08.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:05:08.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:05:08.801 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:05:08.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:05:08.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:05:08.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:05:08.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:05:08.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:05:08.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:05:08.835 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:08.837+0000 7faf0cb27640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:08.838 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:08.841+0000 7faf0cb27640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:08.840 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:08.841+0000 7faf0cb27640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:09.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:05:09.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:05:09.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:05:09.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:05:09.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:05:09.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:09.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:05:09.119 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:05:09.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:09.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:09.383 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:09.385+0000 7faf0cb27640 -1 Falling back to public interface 2024-02-05T05:05:09.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:10.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:10.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:10.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:05:10.449 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:05:10.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:10.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:10.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:11.739 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:11.741+0000 7faf0cb27640 -1 osd.1 28 log_to_monitors true 2024-02-05T05:05:11.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:11.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:11.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:05:11.765 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:05:11.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:11.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:12.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:13.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:13.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:13.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:05:13.141 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:05:13.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:13.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:13.457 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/928624984,v1:127.0.0.1:6811/928624984] [v2:127.0.0.1:6812/928624984,v1:127.0.0.1:6813/928624984] exists,up d615516f-a8df-4af9-97d1-47f28b108d25 2024-02-05T05:05:13.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:05:13.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:05:13.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:05:13.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:05:13.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:05:13.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:05:13.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:05:13.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:05:13.458 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:05:13.458 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:05:13.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:05:13.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:05:13.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:05:13.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:05:13.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:05:13.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:05:13.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:05:13.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:05:13.555 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:05:13.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:05:13.862 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T05:05:13.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:05:13.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:13.863 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:05:14.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-02-05T05:05:14.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-02-05T05:05:14.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702' 2024-02-05T05:05:14.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:14.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:05:14.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-02-05T05:05:14.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-02-05T05:05:14.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-02-05T05:05:14.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:14.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149702 2024-02-05T05:05:14.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:14.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:05:14.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149702 2024-02-05T05:05:14.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:14.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-02-05T05:05:14.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-02-05T05:05:14.166 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 111669149702 2024-02-05T05:05:14.166 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:14.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149702 2024-02-05T05:05:14.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:05:15.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:05:15.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:15.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149702 2024-02-05T05:05:15.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:05:16.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:05:16.788 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:17.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149703 -lt 111669149702 2024-02-05T05:05:17.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:17.094 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-02-05T05:05:17.094 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:17.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:05:17.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-02-05T05:05:17.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:17.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-02-05T05:05:17.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-02-05T05:05:17.098 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 137438953474 2024-02-05T05:05:17.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:05:17.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-02-05T05:05:17.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:05:17.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:17.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:17.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:05:17.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:17.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:17.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:17.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:17.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:17.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:17.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:18.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:05:18.085 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:18.085 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:18.085 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:18.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:05:18.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:05:18.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:05:18.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:05:18.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:05:18.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:05:18.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:05:18.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:05:18.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:05:18.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:05:18.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:05:18.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:05:18.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:05:18.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:05:18.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:05:18.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:05:18.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:05:18.653 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:05:18.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:05:18.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:05:18.655 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:05:18.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:05:18.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:05:18.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:05:18.660 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:05:18.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:05:18.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:05:18.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:05:18.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:05:18.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:05:18.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:05:18.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:05:18.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:05:18.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:05:18.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:05:18.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:05:18.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:05:18.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:05:18.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:05:18.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:05:18.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:05:18.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:05:18.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:05:18.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:05:18.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:05:18.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:05:18.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:05:18.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:05:18.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:05:18.676 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:05:18.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:05:18.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:05:18.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:05:18.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:05:18.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:05:18.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:05:18.679 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:05:18.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:05:18.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:05:18.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:05:18.682 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:05:18.682 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:05:18.682 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:05:18.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:05:18.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:05:18.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:05:18.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:05:18.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:05:18.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:05:18.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:05:18.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T05:05:18.687 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:05:18.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:05:18.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:05:18.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:05:18.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-02-05T05:05:18.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-02-05T05:05:18.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-02-05T05:05:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-02-05T05:05:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-02-05T05:05:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-02-05T05:05:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-02-05T05:05:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-02-05T05:05:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-02-05T05:05:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=1 2024-02-05T05:05:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=testpool 2024-02-05T05:05:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=testpool 2024-02-05T05:05:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-02-05T05:05:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-02-05T05:05:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-02-05T05:05:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-02-05T05:05:20.408 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 7638d430-9701-4d58-8b39-f8896f7dd976 2024-02-05T05:05:20.667 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:05:20.716 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:20.717+0000 7f3a8baa6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:20.718 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:20.721+0000 7f3a8baa6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:20.720 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:20.721+0000 7f3a8baa6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:20.720 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:20.721+0000 7f3a8baa6640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:05:20.720 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:20.721+0000 7f3a8baa6640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:05:20.720 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:20.721+0000 7f3a8baa6640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:05:20.801 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:20.805+0000 7f3a8baa6640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:05:26.657 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:05:27.042 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:05:27.079 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:27.080+0000 7f1ba9aad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:27.093 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:27.096+0000 7f1ba9aad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:27.094 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:27.096+0000 7f1ba9aad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:27.366 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:05:27.651 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:27.652+0000 7f1ba9aad640 -1 Falling back to public interface 2024-02-05T05:05:28.693 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:05:30.009 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:05:30.141 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:30.144+0000 7f1ba9aad640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:05:31.331 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:05:32.680 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:05:34.001 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:05:35.308 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:05:36.610 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:05:38.026 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:05:39.351 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:05:40.688 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:05:42.099 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:05:43.401 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:05:44.891 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:05:46.665 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:05:48.486 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:05:49.878 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:05:51.448 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:05:53.023 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:05:53.329 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2744867401,v1:127.0.0.1:6803/2744867401] [v2:127.0.0.1:6804/2744867401,v1:127.0.0.1:6805/2744867401] exists,up 7638d430-9701-4d58-8b39-f8896f7dd976 2024-02-05T05:05:53.335 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 fdda2e53-6283-4f47-8841-a3b912c8082f 2024-02-05T05:05:53.767 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:05:53.826 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:53.828+0000 7fb2611b7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:53.843 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:53.844+0000 7fb2611b7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:53.844 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:53.844+0000 7fb2611b7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:53.845 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:53.848+0000 7fb2611b7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:05:53.845 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:53.848+0000 7fb2611b7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:05:53.845 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:53.848+0000 7fb2611b7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:05:53.913 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:05:53.916+0000 7fb2611b7640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:05:59.981 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:06:00.357 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:06:00.388 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:00.388+0000 7f7d5b1ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:00.485 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:00.392+0000 7f7d5b1ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:00.485 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:00.392+0000 7f7d5b1ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:00.683 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:06:00.935 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:00.936+0000 7f7d5b1ba640 -1 Falling back to public interface 2024-02-05T05:06:02.000 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:06:03.302 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:06:03.433 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:03.436+0000 7f7d5b1ba640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:06:04.623 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:06:05.980 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:06:07.315 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:06:08.721 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:06:10.033 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:06:11.400 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:06:12.768 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:06:14.122 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:06:15.536 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:06:16.897 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:06:18.482 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:06:20.059 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:06:21.987 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:06:23.519 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:06:25.112 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:06:26.502 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:06:26.818 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/82132515,v1:127.0.0.1:6811/82132515] [v2:127.0.0.1:6812/82132515,v1:127.0.0.1:6813/82132515] exists,up fdda2e53-6283-4f47-8841-a3b912c8082f 2024-02-05T05:06:26.821 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 f472dad9-fe01-4e46-9951-cc1830fabbd0 2024-02-05T05:06:27.211 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:06:27.271 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:27.271+0000 7f8577241640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:27.303 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:27.303+0000 7f8577241640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:27.305 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:27.307+0000 7f8577241640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:27.305 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:27.307+0000 7f8577241640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:06:27.306 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:27.307+0000 7f8577241640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:06:27.306 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:27.307+0000 7f8577241640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:06:27.411 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:27.415+0000 7f8577241640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:06:33.716 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T05:06:34.103 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T05:06:34.140 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:34.139+0000 7fda2053a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:34.160 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:34.159+0000 7fda2053a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:34.162 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:34.163+0000 7fda2053a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:34.417 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:06:34.707 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:34.707+0000 7fda2053a640 -1 Falling back to public interface 2024-02-05T05:06:35.739 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:06:37.039 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:06:37.218 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:06:37.219+0000 7fda2053a640 -1 osd.2 0 log_to_monitors true 2024-02-05T05:06:38.413 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:06:39.725 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:06:41.032 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:06:42.354 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:06:43.675 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:06:45.092 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:06:46.397 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:06:47.716 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:06:49.116 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:06:50.801 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:06:52.780 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:06:54.241 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:06:55.996 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:06:57.690 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:06:59.261 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:07:00.691 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:07:01.004 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2681349561,v1:127.0.0.1:6819/2681349561] [v2:127.0.0.1:6820/2681349561,v1:127.0.0.1:6821/2681349561] exists,up f472dad9-fe01-4e46-9951-cc1830fabbd0 2024-02-05T05:07:01.434 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'testpool' created 2024-02-05T05:07:03.489 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836499 2024-02-05T05:07:05.110 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672973 2024-02-05T05:07:05.441 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509443 2024-02-05T05:07:08.052 INFO:tasks.workunit.client.0.smithi148.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-02-05T05:07:08.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-02-05T05:07:08.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-02-05T05:07:08.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-02-05T05:07:08.450 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 1 size to 2 2024-02-05T05:07:08.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-02-05T05:07:08.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:07:08.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:07:08.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:07:08.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:07:08.469 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:07:08.469 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:07:08.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:07:08.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:07:08.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:07:08.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:07:08.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:07:08.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:07:08.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:07:08.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:07:08.572 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:07:09.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:07:09.114 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:07:09.114 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:07:09.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:07:09.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:09.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:07:09.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-02-05T05:07:09.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-02-05T05:07:09.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-02-05T05:07:09.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:09.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:07:09.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-02-05T05:07:09.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-02-05T05:07:09.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672975' 2024-02-05T05:07:09.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:09.415 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:07:09.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-02-05T05:07:09.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-02-05T05:07:09.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672975 2-64424509445' 2024-02-05T05:07:09.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:09.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-02-05T05:07:09.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:09.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:07:09.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-02-05T05:07:09.562 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:09.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-02-05T05:07:09.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-02-05T05:07:09.563 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836501 2024-02-05T05:07:09.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:07:09.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-02-05T05:07:09.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:07:10.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:07:10.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:07:11.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-02-05T05:07:11.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:11.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-02-05T05:07:11.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:11.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:07:11.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-02-05T05:07:11.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:11.193 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672975 2024-02-05T05:07:11.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-02-05T05:07:11.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-02-05T05:07:11.194 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:07:11.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-02-05T05:07:11.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:11.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-02-05T05:07:11.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:11.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:07:11.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-02-05T05:07:11.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:11.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-02-05T05:07:11.522 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509445 2024-02-05T05:07:11.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-02-05T05:07:11.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:07:11.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509445 2024-02-05T05:07:11.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:07:11.838 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:07:11.838 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:07:12.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:07:12.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:07:12.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:07:12.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:07:12.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:07:12.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:07:12.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:07:12.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:07:12.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:07:12.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:07:12.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:07:12.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:07:12.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:07:12.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:07:12.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:07:12.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-02-05T05:07:12.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-02-05T05:07:12.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:07:13.050 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-02-05T05:07:13.050 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:07:13.051 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:07:13.051 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T05:07:13.051 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:07:13.051 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:07:13.051 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:07:13.051 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:07:13.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:07:13.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:07:13.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:07:13.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:07:13.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:07:13.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:07:13.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:07:13.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:07:13.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:07:13.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:07:13.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:07:13.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:07:13.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:07:13.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:07:13.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:07:13.676 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:07:13.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:07:13.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:07:13.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:07:13.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:07:13.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:07:13.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:07:13.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:07:13.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:07:13.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:07:13.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:07:13.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-02-05T05:07:15.266 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-02-05T05:07:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:07:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:07:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:07:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:07:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:07:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:07:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:07:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:07:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:07:15.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:07:15.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:07:15.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:07:15.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:07:15.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:07:15.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:07:15.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:07:15.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:07:15.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:07:15.815 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:07:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:07:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:07:15.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:07:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:07:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:07:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:07:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:07:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:07:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:07:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:07:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:07:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:07:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:07:15.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:07:15.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:07:15.817 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:07:15.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:07:15.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:07:15.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:07:15.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:07:15.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:07:15.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:07:15.856 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:15.858+0000 7f5733600640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:15.861 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:15.866+0000 7f5733600640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:15.862 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:15.866+0000 7f5733600640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:07:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:07:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:07:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:07:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:07:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:07:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:07:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:16.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:07:16.399 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:16.402+0000 7f5733600640 -1 Falling back to public interface 2024-02-05T05:07:16.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:17.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:17.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:17.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:07:17.484 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:07:17.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:17.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:07:17.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:18.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:18.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:18.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:07:18.791 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:07:18.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:18.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:07:18.939 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:18.942+0000 7f5733600640 -1 osd.0 21 log_to_monitors true 2024-02-05T05:07:19.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:20.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:20.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:20.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:07:20.143 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:07:20.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:20.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:07:20.465 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1080186546,v1:127.0.0.1:6803/1080186546] [v2:127.0.0.1:6804/1080186546,v1:127.0.0.1:6805/1080186546] exists,up 7638d430-9701-4d58-8b39-f8896f7dd976 2024-02-05T05:07:20.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:07:20.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:07:20.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:07:20.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:07:20.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:07:20.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:07:20.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:07:20.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:07:20.474 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:07:20.474 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:07:20.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:07:20.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:07:20.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:07:20.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:07:20.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:07:20.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:07:20.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:07:20.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:07:20.573 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:07:20.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:07:20.871 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:07:20.872 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:07:20.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:07:20.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:20.872 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:07:21.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-02-05T05:07:21.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-02-05T05:07:21.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-02-05T05:07:21.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:21.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:07:21.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-02-05T05:07:21.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-02-05T05:07:21.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978' 2024-02-05T05:07:21.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:21.194 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:07:21.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-02-05T05:07:21.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-02-05T05:07:21.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978 2-64424509449' 2024-02-05T05:07:21.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:21.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-02-05T05:07:21.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:21.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:07:21.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-02-05T05:07:21.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:21.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-02-05T05:07:21.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-02-05T05:07:21.338 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 107374182402 2024-02-05T05:07:21.338 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:07:21.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-02-05T05:07:21.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:07:22.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:07:22.644 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:07:22.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-02-05T05:07:22.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:07:23.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:07:23.969 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:07:24.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-02-05T05:07:24.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:24.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-02-05T05:07:24.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:24.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:07:24.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-02-05T05:07:24.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:24.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-02-05T05:07:24.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-02-05T05:07:24.290 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672978 2024-02-05T05:07:24.290 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:07:24.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672978 2024-02-05T05:07:24.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:24.596 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-02-05T05:07:24.596 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:24.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:07:24.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-02-05T05:07:24.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:24.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-02-05T05:07:24.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-02-05T05:07:24.599 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509449 2024-02-05T05:07:24.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:07:24.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-02-05T05:07:24.920 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:07:24.920 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:07:24.921 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:07:25.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:07:25.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:07:25.313 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:07:25.313 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:07:25.313 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:07:25.313 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:07:25.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:07:25.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:07:25.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:07:25.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:07:25.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:07:25.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:07:25.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:07:25.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:07:25.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:07:25.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-02-05T05:07:26.115 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T05:07:26.115 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T05:07:26.115 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T05:07:26.127 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T05:07:26.127 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-02-05T05:07:26.127 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T05:07:26.139 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2: { 2024-02-05T05:07:26.139 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-02-05T05:07:26.140 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T05:07:26.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-02-05T05:07:26.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-02-05T05:07:26.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:07:26.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-02-05T05:07:26.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:07:26.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-02-05T05:07:26.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-02-05T05:07:26.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:07:26.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:07:26.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:07:26.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:07:26.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-02-05T05:07:26.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-02-05T05:07:26.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:07:26.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:07:26.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:07:26.753 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:07:27.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-02-05T05:07:01.370091+0000 2024-02-05T05:07:27.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-02-05T05:07:27.210 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T05:07:27.210 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": true, 2024-02-05T05:07:27.210 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T05:07:27.210 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-22T05:05:47.215316+0000" 2024-02-05T05:07:27.210 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T05:07:27.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-02-05T05:07:01.370091+0000 2024-02-05T05:07:27.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T05:07:27.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:07:01.370091+0000 2024-02-05T05:07:27.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:07:27.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:07:27.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:07:27.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:07:27.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:07:27.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:07:27.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:07:27.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:07:27.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:07:01.370091+0000 '>' 2024-02-05T05:07:01.370091+0000 2024-02-05T05:07:27.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:07:28.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:07:28.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:07:28.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:07:28.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:07:28.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:07:28.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:07:28.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:07:28.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:07:01.370091+0000 '>' 2024-02-05T05:07:01.370091+0000 2024-02-05T05:07:28.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:07:29.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:07:29.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:07:29.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:07:29.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:07:29.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:07:29.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:07:29.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:07:30.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:07:01.370091+0000 '>' 2024-02-05T05:07:01.370091+0000 2024-02-05T05:07:30.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:07:31.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:07:31.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:07:31.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:07:31.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:07:31.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:07:31.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:07:31.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:07:31.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:07:01.370091+0000 '>' 2024-02-05T05:07:01.370091+0000 2024-02-05T05:07:31.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:07:32.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:07:32.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:07:32.451 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:07:32.451 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:07:32.451 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:07:32.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:07:32.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:07:32.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:07:01.370091+0000 '>' 2024-02-05T05:07:01.370091+0000 2024-02-05T05:07:32.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:07:33.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:07:33.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:07:33.767 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:07:33.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:07:33.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:07:33.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:07:33.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:07:34.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:07:28.004606+0000 '>' 2024-02-05T05:07:01.370091+0000 2024-02-05T05:07:34.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:07:34.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-02-05T05:07:34.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:07:34.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:07:34.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:07:34.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:07:34.084 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:07:34.084 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:07:34.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:07:34.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:07:34.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:07:34.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:07:34.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:07:34.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:07:34.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:07:34.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:07:34.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:07:34.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:07:34.513 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:07:34.513 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:07:34.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:07:34.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:34.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:07:34.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182406 2024-02-05T05:07:34.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182406 2024-02-05T05:07:34.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406' 2024-02-05T05:07:34.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:34.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:07:34.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-02-05T05:07:34.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-02-05T05:07:34.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-42949672982' 2024-02-05T05:07:34.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:34.811 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:07:34.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-02-05T05:07:34.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-02-05T05:07:34.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-42949672982 2-64424509452' 2024-02-05T05:07:34.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:34.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182406 2024-02-05T05:07:34.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:34.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:07:34.953 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182406 2024-02-05T05:07:34.953 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:34.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182406 2024-02-05T05:07:34.954 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 107374182406 2024-02-05T05:07:34.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2024-02-05T05:07:34.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:07:35.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182406 2024-02-05T05:07:35.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:35.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-02-05T05:07:35.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:35.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:07:35.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-02-05T05:07:35.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:35.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-02-05T05:07:35.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-02-05T05:07:35.266 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672982 2024-02-05T05:07:35.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:07:35.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-02-05T05:07:35.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:35.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-02-05T05:07:35.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:35.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:07:35.570 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-02-05T05:07:35.570 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:35.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-02-05T05:07:35.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-02-05T05:07:35.571 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509452 2024-02-05T05:07:35.571 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:07:35.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509452 2024-02-05T05:07:35.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:07:36.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:07:36.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:07:37.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509452 2024-02-05T05:07:37.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:07:37.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:07:37.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:07:37.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:07:37.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:07:37.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:07:37.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:07:37.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:07:37.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:07:37.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:07:37.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:07:37.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:07:37.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:07:37.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:07:37.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:07:38.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:07:38.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:07:38.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:07:38.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-02-05T05:07:38.530 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T05:07:38.531 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-02-05T05:07:28.163710+0000 21'1 26:57 [1,0] 1 [1,0] 1 21'1 2024-02-05T05:07:28.004606+0000 21'1 2024-02-05T05:07:28.004606+0000 0 1 periodic scrub scheduled @ 2024-02-06T05:07:28.004606+0000 1 0 2024-02-05T05:07:38.531 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T05:07:38.531 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T05:07:38.531 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:07:38.548 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-02-05T05:07:38.548 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:07:38.548 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:07:38.549 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T05:07:38.549 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:07:38.549 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:07:38.549 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:07:38.549 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:07:38.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:07:38.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:07:38.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:07:39.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T05:07:39.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:07:39.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:07:39.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:07:39.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:07:39.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T05:07:39.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:07:39.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:07:39.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:07:39.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:07:39.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:07:39.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:07:39.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:07:39.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:07:39.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:07:39.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:07:39.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:07:39.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T05:07:39.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:07:39.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:07:39.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:07:39.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:07:39.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:07:39.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-02-05T05:07:40.661 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:07:40.661 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:07:41.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:07:41.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:07:41.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:07:41.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:07:41.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:07:41.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:07:41.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:07:41.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:07:41.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:07:41.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:07:41.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:07:41.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:07:41.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:07:41.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:07:41.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:07:41.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:07:41.198 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:07:41.198 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:07:41.198 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:07:41.198 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:07:41.198 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:07:41.199 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:07:41.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:07:41.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:07:41.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:07:41.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:07:41.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:07:41.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:07:41.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:07:41.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:07:41.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:07:41.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:07:41.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:07:41.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:07:41.200 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:07:41.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:07:41.200 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:07:41.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:07:41.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:07:41.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:07:41.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:07:41.234 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:41.237+0000 7f794f3b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:41.236 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:41.241+0000 7f794f3b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:41.238 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:41.241+0000 7f794f3b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:41.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:07:41.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:07:41.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:07:41.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:07:41.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:07:41.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:41.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:07:41.524 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:07:41.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:41.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:07:41.771 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:41.773+0000 7f794f3b8640 -1 Falling back to public interface 2024-02-05T05:07:41.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:42.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:42.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:42.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:07:42.889 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:07:42.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:42.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:07:43.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:44.120 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:44.121+0000 7f794f3b8640 -1 osd.0 26 log_to_monitors true 2024-02-05T05:07:44.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:44.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:44.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:07:44.216 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:07:44.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:44.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:07:44.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:45.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:45.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:45.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:07:45.517 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:07:45.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:45.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:07:45.868 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/4112574016,v1:127.0.0.1:6803/4112574016] [v2:127.0.0.1:6804/4112574016,v1:127.0.0.1:6805/4112574016] exists,up 7638d430-9701-4d58-8b39-f8896f7dd976 2024-02-05T05:07:45.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:07:45.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:07:45.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:07:45.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:07:45.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:07:45.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:07:45.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:07:45.869 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:07:45.871 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:07:45.871 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:07:45.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:07:45.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:07:45.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:07:45.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:07:45.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:07:45.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:07:45.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:07:45.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:07:45.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:07:46.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:07:46.285 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:07:46.285 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:07:46.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:07:46.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:46.286 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:07:46.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-02-05T05:07:46.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-02-05T05:07:46.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-02-05T05:07:46.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:46.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:07:46.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-02-05T05:07:46.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-02-05T05:07:46.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672985' 2024-02-05T05:07:46.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:46.587 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:07:46.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-02-05T05:07:46.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-02-05T05:07:46.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672985 2-64424509456' 2024-02-05T05:07:46.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:46.728 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-02-05T05:07:46.728 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:46.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:07:46.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-02-05T05:07:46.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:46.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-02-05T05:07:46.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-02-05T05:07:46.730 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 128849018882 2024-02-05T05:07:46.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:07:47.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-02-05T05:07:47.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:47.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-02-05T05:07:47.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:47.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:07:47.048 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-02-05T05:07:47.048 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:47.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-02-05T05:07:47.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-02-05T05:07:47.050 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672985 2024-02-05T05:07:47.050 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:07:47.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672985 -lt 42949672985 2024-02-05T05:07:47.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:47.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-02-05T05:07:47.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:47.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:07:47.386 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-02-05T05:07:47.386 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:47.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-02-05T05:07:47.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-02-05T05:07:47.388 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509456 2024-02-05T05:07:47.389 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:07:47.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-02-05T05:07:47.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:07:47.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:07:47.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:07:48.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:07:48.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:07:48.076 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:07:48.076 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:07:48.076 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:07:48.076 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:07:48.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:07:48.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:07:48.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:07:48.375 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:07:48.375 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:07:48.375 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:07:48.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:07:48.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:07:48.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:07:48.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-02-05T05:07:48.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:07:48.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:07:48.761 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T05:07:48.761 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:07:48.761 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:07:48.761 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:07:48.761 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:07:49.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:07:49.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:07:49.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:07:49.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T05:07:49.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:07:49.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:07:49.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:07:49.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:07:49.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T05:07:49.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:07:49.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:07:49.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:07:49.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:07:49.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:07:49.391 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:07:49.391 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:07:49.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:07:49.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:07:49.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:07:49.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:07:49.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T05:07:49.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:07:49.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:07:49.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:07:49.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:07:49.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:07:49.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T05:07:51.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:07:51.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:07:51.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:07:51.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:07:51.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:07:51.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:07:51.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:07:51.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:07:51.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:07:51.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:07:51.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:07:51.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:07:51.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:07:51.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:07:51.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:07:51.166 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:07:51.166 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:07:51.166 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:07:51.166 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:07:51.166 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:07:51.166 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:07:51.166 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:07:51.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:07:51.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:07:51.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:07:51.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:07:51.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:07:51.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:07:51.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:07:51.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:07:51.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:07:51.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:07:51.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:07:51.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:07:51.168 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:07:51.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:07:51.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:07:51.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:07:51.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:07:51.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:07:51.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:07:51.207 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:51.209+0000 7f47cc722640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:51.210 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:51.213+0000 7f47cc722640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:51.211 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:51.213+0000 7f47cc722640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:51.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:07:51.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:07:51.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:07:51.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:07:51.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:07:51.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:51.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:07:51.486 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:07:51.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:51.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:07:51.747 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:51.749+0000 7f47cc722640 -1 Falling back to public interface 2024-02-05T05:07:51.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:52.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:52.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:52.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:07:52.791 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:07:52.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:52.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:07:53.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:54.042 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:07:54.045+0000 7f47cc722640 -1 osd.0 31 log_to_monitors true 2024-02-05T05:07:54.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:54.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:54.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:07:54.135 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:07:54.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:54.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:07:54.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:55.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:55.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:55.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:07:55.513 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:07:55.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:55.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:07:55.841 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/4034510034,v1:127.0.0.1:6803/4034510034] [v2:127.0.0.1:6804/4034510034,v1:127.0.0.1:6805/4034510034] exists,up 7638d430-9701-4d58-8b39-f8896f7dd976 2024-02-05T05:07:55.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:07:55.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:07:55.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:07:55.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:07:55.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:07:55.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:07:55.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:07:55.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:07:55.843 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:07:55.843 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:07:55.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:07:55.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:07:55.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:07:55.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:07:55.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:07:55.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:07:55.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:07:55.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:07:55.940 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:07:56.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:07:56.270 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:07:56.270 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:07:56.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:07:56.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:56.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:07:56.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-02-05T05:07:56.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-02-05T05:07:56.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-02-05T05:07:56.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:56.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:07:56.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672988 2024-02-05T05:07:56.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672988 2024-02-05T05:07:56.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672988' 2024-02-05T05:07:56.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:07:56.578 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:07:56.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-02-05T05:07:56.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-02-05T05:07:56.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672988 2-64424509459' 2024-02-05T05:07:56.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:56.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-02-05T05:07:56.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:56.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:07:56.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-02-05T05:07:56.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:56.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-02-05T05:07:56.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-02-05T05:07:56.722 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 150323855362 2024-02-05T05:07:56.722 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:07:57.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-02-05T05:07:57.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:57.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672988 2024-02-05T05:07:57.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:57.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:07:57.018 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:57.018 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672988 2024-02-05T05:07:57.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672988 2024-02-05T05:07:57.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-02-05T05:07:57.020 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672988 2024-02-05T05:07:57.020 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:07:57.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672988 2024-02-05T05:07:57.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:07:57.313 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-02-05T05:07:57.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:07:57.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:07:57.316 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-02-05T05:07:57.316 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:07:57.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-02-05T05:07:57.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-02-05T05:07:57.317 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509459 2024-02-05T05:07:57.318 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:07:57.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-02-05T05:07:57.621 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:07:57.623 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:07:57.623 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:07:58.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:07:58.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:07:58.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:07:58.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:07:58.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:07:58.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:07:58.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:07:58.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:07:58.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:07:58.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:07:58.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:07:58.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:07:58.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:07:58.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:07:58.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:07:58.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:07:58.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-02-05T05:07:58.693 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:07:28.002+0000 7f7d3f144640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T05:07:58.693 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:07:28.002+0000 7f7d3f144640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-02-05T05:07:58.693 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:07:28.002+0000 7f7d3f144640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-02-05T05:07:58.693 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:07:28.002+0000 7f7d3f144640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-02-05T05:07:58.694 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:07:28.002+0000 7f7d3f144640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T05:07:58.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:07:58.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:07:58.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:07:58.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:07:58.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:07:58.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:07:58.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:07:58.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:07:58.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:07:59.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:07:59.005 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:07:59.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:07:59.007 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:07:59.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:07:59.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:07:59.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:07:59.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:07:59.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:07:59.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:07:59.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:07:59.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:07:59.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:07:59.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:07:59.037 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:07:59.037 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:07:59.037 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:07:59.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:07:59.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:07:59.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:07:59.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:07:59.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:07:59.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:07:59.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:07:59.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:07:59.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:07:59.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:07:59.038 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:07:59.038 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:07:59.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:07:59.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:07:59.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:07:59.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:07:59.039 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:07:59.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:07:59.040 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:07:59.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:07:59.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:07:59.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:07:59.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:07:59.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:07:59.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:07:59.043 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:07:59.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:07:59.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:07:59.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:07:59.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:07:59.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:07:59.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:07:59.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:07:59.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:07:59.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:07:59.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:07:59.050 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:07:59.050 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:07:59.050 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:07:59.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T05:07:59.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:07:59.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:07:59.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:07:59.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:07:59.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-02-05T05:07:59.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-02-05T05:07:59.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-02-05T05:07:59.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-02-05T05:07:59.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:07:59.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:07:59.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:07:59.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:07:59.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:07:59.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:07:59.442 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:07:59.442 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:07:59.442 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:07:59.443 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:07:59.443 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:07:59.443 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:07:59.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:07:59.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:07:59.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:07:59.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:07:59.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:07:59.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:07:59.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:07:59.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:07:59.639 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:07:59.639 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:07:59.640 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:07:59.640 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:07:59.640 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:07:59.640 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:07:59.640 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:07:59.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:07:59.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T05:07:59.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:07:59.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:07:59.756 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:07:59.756 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:07:59.757 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:07:59.757 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:07:59.757 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:07:59.757 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:07:59.757 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:07:59.757 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:07:59.757 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:07:59.757 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:07:59.758 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:07:59.758 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T05:07:59.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-02-05T05:07:59.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:07:59.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:07:59.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:07:59.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:07:59.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:07:59.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:08:00.738 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:08:00.738 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:08:00.738 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:08:00.739 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:08:00.739 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:08:00.739 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:08:00.739 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:08:00.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:08:00.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:08:00.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:08:00.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-02-05T05:08:00.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-02-05T05:08:00.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:08:00.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:08:00.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:08:00.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:08:00.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:08:00.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:08:00.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:08:00.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:08:00.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:08:00.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:08:00.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:08:00.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:08:00.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:08:00.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:08:00.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:08:00.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:08:00.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:08:00.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:08:00.779 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:08:00.779 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:08:00.779 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:08:00.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:08:00.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:08:00.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:08:00.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:08:00.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:08:00.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:08:00.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:08:00.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:08:00.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:08:00.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:08:00.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:08:00.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:08:00.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:08:00.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:08:00.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:08:00.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=195f6dc1-aa07-4514-ada5-2223daea6e28 2024-02-05T05:08:00.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 195f6dc1-aa07-4514-ada5-2223daea6e28' 2024-02-05T05:08:00.782 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 195f6dc1-aa07-4514-ada5-2223daea6e28 2024-02-05T05:08:00.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:08:00.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwbcBlrpPkLxAALZ7BE1xkmnYmr5ohBU2/ZA== 2024-02-05T05:08:00.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwbcBlrpPkLxAALZ7BE1xkmnYmr5ohBU2/ZA=="}' 2024-02-05T05:08:00.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 195f6dc1-aa07-4514-ada5-2223daea6e28 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:08:01.040 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:08:01.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:08:01.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAwbcBlrpPkLxAALZ7BE1xkmnYmr5ohBU2/ZA== --osd-uuid 195f6dc1-aa07-4514-ada5-2223daea6e28 2024-02-05T05:08:01.095 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:01.097+0000 7f6ad3c12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:01.097 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:01.101+0000 7f6ad3c12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:01.099 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:01.101+0000 7f6ad3c12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:01.099 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:01.101+0000 7f6ad3c12640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:08:01.099 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:01.101+0000 7f6ad3c12640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:08:01.099 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:01.101+0000 7f6ad3c12640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:08:01.183 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:01.185+0000 7f6ad3c12640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:08:07.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:08:07.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:08:07.060 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:08:07.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:08:07.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:08:07.476 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:08:07.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:08:07.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:08:07.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:08:07.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:08:07.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:08:07.510 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:07.513+0000 7fc0ae5b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:07.512 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:07.513+0000 7fc0ae5b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:07.513 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:07.517+0000 7fc0ae5b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:07.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:08:07.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:08:07.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:08:07.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:08:07.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:08:07.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:07.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:08:07.798 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:08:07.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:07.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:08.062 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:08.065+0000 7fc0ae5b8640 -1 Falling back to public interface 2024-02-05T05:08:08.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:09.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:09.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:09.101 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:08:09.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:08:09.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:09.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:09.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:10.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:10.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:10.423 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:08:10.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:08:10.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:10.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:10.525 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:10.529+0000 7fc0ae5b8640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:08:10.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:11.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:11.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:11.791 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:08:11.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:08:11.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:11.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:12.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:13.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:13.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:13.112 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:08:13.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:08:13.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:13.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:13.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:14.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:14.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:14.528 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:08:14.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:08:14.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:14.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:14.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:15.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:15.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:15.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:08:15.838 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:08:15.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:15.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:16.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:17.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:17.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:17.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:08:17.251 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:08:17.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:17.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:17.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:18.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:18.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:18.557 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:08:18.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:08:18.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:18.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:18.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:19.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:19.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:19.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:08:19.862 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:08:19.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:19.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:20.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:21.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:21.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:21.297 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:08:21.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:08:21.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:21.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:21.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:22.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:22.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:22.617 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:08:22.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:08:22.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:22.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:22.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:23.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:23.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:23.935 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:08:23.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:08:23.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:23.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:24.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:25.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:25.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:25.684 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:08:25.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:08:25.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:25.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:26.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:27.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:27.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:27.334 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:08:27.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:08:27.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:27.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:28.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:29.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:29.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:29.308 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:08:29.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:08:29.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:29.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:29.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:30.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:30.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:30.793 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:08:30.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:08:30.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:30.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:31.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:32.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:32.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:32.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:08:32.265 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:08:32.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:32.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:32.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:33.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:33.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:33.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:08:33.662 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:08:33.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:33.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:08:33.982 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3397132645,v1:127.0.0.1:6803/3397132645] [v2:127.0.0.1:6804/3397132645,v1:127.0.0.1:6805/3397132645] exists,up 195f6dc1-aa07-4514-ada5-2223daea6e28 2024-02-05T05:08:33.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:08:33.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:08:33.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:08:33.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-02-05T05:08:33.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:08:33.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:08:33.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:08:33.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:08:33.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:08:33.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:08:33.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:08:33.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:08:33.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:08:33.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:08:33.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:08:33.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:08:33.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:08:33.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:08:33.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:08:33.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:08:33.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:08:33.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:08:33.987 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:08:33.988 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:08:33.988 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:08:33.988 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:08:33.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:08:33.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:08:33.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:08:33.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:08:33.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:08:33.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:08:33.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:08:33.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:08:33.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:08:33.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:08:33.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:08:33.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:08:33.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:08:33.989 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:08:33.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5838c5b9-eeba-4a7f-a55b-62a337768543 2024-02-05T05:08:33.989 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 5838c5b9-eeba-4a7f-a55b-62a337768543 2024-02-05T05:08:33.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5838c5b9-eeba-4a7f-a55b-62a337768543' 2024-02-05T05:08:33.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:08:34.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSbcBlwbEFARAA1hT4poH0uEQVulQ25aeQ4Q== 2024-02-05T05:08:34.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSbcBlwbEFARAA1hT4poH0uEQVulQ25aeQ4Q=="}' 2024-02-05T05:08:34.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5838c5b9-eeba-4a7f-a55b-62a337768543 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:08:34.366 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:08:34.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:08:34.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBSbcBlwbEFARAA1hT4poH0uEQVulQ25aeQ4Q== --osd-uuid 5838c5b9-eeba-4a7f-a55b-62a337768543 2024-02-05T05:08:34.415 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:34.416+0000 7fc1f49a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:34.451 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:34.452+0000 7fc1f49a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:34.452 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:34.456+0000 7fc1f49a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:34.453 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:34.456+0000 7fc1f49a2640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:08:34.453 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:34.456+0000 7fc1f49a2640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:08:34.453 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:34.456+0000 7fc1f49a2640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:08:34.517 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:34.520+0000 7fc1f49a2640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:08:40.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:08:40.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:08:40.655 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:08:40.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:08:40.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:08:41.047 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:08:41.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:08:41.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:08:41.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:08:41.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:08:41.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:08:41.082 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:41.084+0000 7fd1b20da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:41.100 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:41.100+0000 7fd1b20da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:41.101 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:41.104+0000 7fd1b20da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:41.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:08:41.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:08:41.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:08:41.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:08:41.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:08:41.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:41.368 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:08:41.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:08:41.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:41.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:08:41.659 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:41.660+0000 7fd1b20da640 -1 Falling back to public interface 2024-02-05T05:08:41.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:42.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:42.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:42.678 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:08:42.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:08:42.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:42.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:08:42.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:43.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:43.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:43.986 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:08:43.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:08:43.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:43.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:08:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:44.318 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:08:44.320+0000 7fd1b20da640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:08:45.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:45.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:45.304 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:08:45.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:08:45.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:45.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:08:45.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:46.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:46.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:46.670 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:08:46.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:08:46.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:46.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:08:46.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:47.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:47.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:47.990 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:08:47.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:08:47.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:47.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:08:48.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:49.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:49.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:49.311 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:08:49.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:08:49.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:49.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:08:49.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:50.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:50.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:50.629 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:08:50.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:08:50.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:50.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:08:50.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:51.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:51.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:51.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:08:51.948 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:08:51.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:51.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:08:52.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:53.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:53.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:53.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:08:53.335 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:08:53.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:53.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:08:53.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:54.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:54.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:54.648 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:08:54.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:08:54.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:54.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:08:54.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:55.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:55.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:55.952 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:08:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:08:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:08:56.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:57.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:57.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:57.274 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:08:57.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:08:57.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:57.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:08:58.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:59.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:59.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:59.240 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:08:59.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:08:59.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:59.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:08:59.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:00.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:00.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:00.612 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:09:00.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:09:00.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:00.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:09:01.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:02.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:02.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:02.358 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:09:02.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:09:02.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:02.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:09:03.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:04.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:04.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:04.076 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:09:04.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:09:04.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:04.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:09:04.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:05.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:05.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:05.874 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:09:05.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:09:05.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:05.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:09:06.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:07.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:07.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:07.410 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:09:07.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:09:07.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:07.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:09:07.727 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3534676072,v1:127.0.0.1:6811/3534676072] [v2:127.0.0.1:6812/3534676072,v1:127.0.0.1:6813/3534676072] exists,up 5838c5b9-eeba-4a7f-a55b-62a337768543 2024-02-05T05:09:07.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:09:07.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:09:07.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:09:07.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-02-05T05:09:07.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:09:07.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:09:07.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:09:07.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:09:07.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:09:07.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:09:07.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:09:07.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:09:07.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:09:07.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:09:07.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:09:07.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:09:07.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:09:07.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:09:07.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:09:07.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:09:07.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:09:07.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:09:07.731 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:09:07.731 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:09:07.731 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:09:07.732 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:09:07.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:09:07.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:09:07.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:09:07.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:09:07.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:09:07.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:09:07.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:09:07.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:09:07.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:09:07.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:09:07.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:09:07.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:09:07.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:09:07.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:09:07.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=42556976-b380-4d3c-9c96-e8b8a6789b60 2024-02-05T05:09:07.734 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 42556976-b380-4d3c-9c96-e8b8a6789b60 2024-02-05T05:09:07.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 42556976-b380-4d3c-9c96-e8b8a6789b60' 2024-02-05T05:09:07.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:09:07.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzbcBllF5RLRAAD5o8W9Y15w874ptpTJnWZg== 2024-02-05T05:09:07.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzbcBllF5RLRAAD5o8W9Y15w874ptpTJnWZg=="}' 2024-02-05T05:09:07.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 42556976-b380-4d3c-9c96-e8b8a6789b60 -i td/osd-scrub-repair/2/new.json 2024-02-05T05:09:08.110 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:09:08.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:09:08.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBzbcBllF5RLRAAD5o8W9Y15w874ptpTJnWZg== --osd-uuid 42556976-b380-4d3c-9c96-e8b8a6789b60 2024-02-05T05:09:08.167 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:08.168+0000 7f29d7306640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:08.177 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:08.180+0000 7f29d7306640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:08.178 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:08.180+0000 7f29d7306640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:08.179 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:08.180+0000 7f29d7306640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:09:08.179 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:08.180+0000 7f29d7306640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:09:08.179 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:08.180+0000 7f29d7306640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:09:08.254 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:08.256+0000 7f29d7306640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:09:14.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:09:14.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:09:14.267 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T05:09:14.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:09:14.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:09:14.640 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T05:09:14.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:09:14.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:09:14.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:09:14.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:09:14.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:09:14.673 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:14.676+0000 7fe5c1f94640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:14.693 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:14.696+0000 7fe5c1f94640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:14.694 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:14.696+0000 7fe5c1f94640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:14.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:09:14.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:09:14.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:09:14.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:09:14.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:09:14.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:14.959 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:09:14.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:09:14.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:14.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:15.246 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:15.248+0000 7fe5c1f94640 -1 Falling back to public interface 2024-02-05T05:09:15.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:16.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:16.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:16.273 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:09:16.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:09:16.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:16.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:16.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:17.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:17.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:17.570 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:09:17.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:09:17.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:17.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:17.779 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:17.780+0000 7fe5c1f94640 -1 osd.2 0 log_to_monitors true 2024-02-05T05:09:17.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:18.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:18.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:18.888 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:09:18.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:09:18.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:18.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:19.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:20.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:20.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:20.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:09:20.208 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:09:20.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:20.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:20.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:21.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:21.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:21.699 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:09:21.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:09:21.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:21.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:22.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:23.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:23.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:23.083 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:09:23.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:09:23.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:23.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:23.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:24.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:24.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:24.406 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:09:24.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:09:24.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:24.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:24.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:25.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:25.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:25.807 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:09:25.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:09:25.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:25.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:26.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:27.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:27.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:27.140 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:09:27.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:09:27.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:27.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:27.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:28.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:28.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:28.680 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:09:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:09:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:28.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:28.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:29.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:29.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:29.997 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:09:29.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:09:29.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:29.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:30.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:31.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:31.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:31.319 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:09:31.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:09:31.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:31.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:31.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:32.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:32.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:32.965 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:09:32.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:09:32.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:32.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:33.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:34.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:34.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:34.453 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:09:34.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:09:34.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:34.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:35.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:36.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:36.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:36.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:09:36.241 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:09:36.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:36.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:36.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:37.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:37.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:37.676 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:09:37.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:09:37.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:37.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:38.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:39.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:39.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:39.447 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:09:39.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:09:39.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:39.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:40.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:41.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:41.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:41.150 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:09:41.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:09:41.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:41.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:41.485 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2122630098,v1:127.0.0.1:6819/2122630098] [v2:127.0.0.1:6820/2122630098,v1:127.0.0.1:6821/2122630098] exists,up 42556976-b380-4d3c-9c96-e8b8a6789b60 2024-02-05T05:09:41.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:09:41.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:09:41.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:09:41.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-02-05T05:09:41.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-02-05T05:09:42.028 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'testpool' created 2024-02-05T05:09:42.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:09:43.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-02-05T05:09:43.558 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 1 size to 2 2024-02-05T05:09:43.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-02-05T05:09:43.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:09:43.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:09:43.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:09:43.580 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:09:43.582 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:09:43.582 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:09:43.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:09:43.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:09:43.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:09:43.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:09:43.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:09:43.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:09:43.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:09:43.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:09:43.704 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:09:44.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:09:44.065 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:09:44.065 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:09:44.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:09:44.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:44.066 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:09:44.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-02-05T05:09:44.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-02-05T05:09:44.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-02-05T05:09:44.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:44.383 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:09:44.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-02-05T05:09:44.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-02-05T05:09:44.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672972' 2024-02-05T05:09:44.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:44.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:09:44.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-02-05T05:09:44.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-02-05T05:09:44.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672972 2-64424509453' 2024-02-05T05:09:44.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:44.679 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-02-05T05:09:44.679 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:44.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:09:44.681 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-02-05T05:09:44.681 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:44.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-02-05T05:09:44.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-02-05T05:09:44.683 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836499 2024-02-05T05:09:44.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:09:45.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-02-05T05:09:45.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:09:46.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:09:46.031 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:09:46.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-02-05T05:09:46.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:46.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-02-05T05:09:46.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:46.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:09:46.347 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-02-05T05:09:46.347 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:46.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-02-05T05:09:46.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-02-05T05:09:46.348 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672972 2024-02-05T05:09:46.348 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:09:46.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-02-05T05:09:46.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:46.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-02-05T05:09:46.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:46.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:09:46.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-02-05T05:09:46.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:46.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-02-05T05:09:46.692 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509453 2024-02-05T05:09:46.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-02-05T05:09:46.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:09:46.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-02-05T05:09:46.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:09:46.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:09:46.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:09:47.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:09:47.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:09:47.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:09:47.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:09:47.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:09:47.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:09:47.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:09:47.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:09:47.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T05:09:47.703 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:09:47.703 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:09:47.704 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:09:48.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T05:09:48.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-05T05:09:48.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:09:48.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-05T05:09:48.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:09:48.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:09:48.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:09:48.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:09:48.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:09:48.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:09:48.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:09:48.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:09:48.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:09:48.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:09:48.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T05:09:48.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:09:48.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:09:48.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:09:48.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T05:09:48.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T05:09:48.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:09:48.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:09:48.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:09:48.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:09:48.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:09:48.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:09:48.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:09:49.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:09:49.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:09:49.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:09:49.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:09:49.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:09:49.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:09:49.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:09:49.380 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:09:49.380 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:09:49.380 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:09:49.380 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:09:49.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:09:49.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:09:49.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:09:49.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:09:49.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:09:49.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:09:50.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:09:50.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:09:50.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:09:50.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-02-05T05:09:50.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-02-05T05:09:50.022 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-02-05T05:09:50.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T05:09:50.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:09:50.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T05:09:50.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:09:50.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T05:09:50.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:09:50.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T05:09:50.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:09:50.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T05:09:50.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:09:50.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T05:09:50.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:09:50.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T05:09:50.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:09:50.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T05:09:50.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:09:51.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T05:09:51.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:09:51.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T05:09:51.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:09:51.428 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-02-05T05:09:51.428 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:09:51.428 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:09:51.428 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T05:09:51.428 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:09:51.429 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:09:51.429 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:09:51.429 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:09:51.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:09:51.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:09:51.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:09:52.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-02-05T05:09:52.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:09:52.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:09:52.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:09:52.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:09:52.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-02-05T05:09:52.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:09:52.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:09:52.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:09:52.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:09:52.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:09:52.074 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:09:52.074 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:09:52.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:09:52.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:09:52.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:09:52.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:09:52.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-02-05T05:09:52.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:09:52.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:09:52.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:09:52.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:09:52.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:09:52.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-02-05T05:09:53.690 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:ff7b1f36:::obj1:head# 2024-02-05T05:09:54.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:09:54.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:09:54.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:09:54.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:09:54.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:09:54.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:09:54.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:09:54.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:09:54.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:09:54.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:09:54.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:09:54.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:09:54.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:09:54.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:09:54.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:09:54.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:09:54.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:09:54.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:09:54.250 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:09:54.250 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:09:54.250 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:09:54.250 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:09:54.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:09:54.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:09:54.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:09:54.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:09:54.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:09:54.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:09:54.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:09:54.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:09:54.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:09:54.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:09:54.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:09:54.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:09:54.251 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:09:54.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:09:54.252 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:09:54.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:09:54.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:09:54.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:09:54.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:09:54.293 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:54.295+0000 7f8920049640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:54.296 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:54.299+0000 7f8920049640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:54.298 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:54.299+0000 7f8920049640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:54.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:09:54.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:09:54.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:09:54.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:09:54.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:09:54.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:54.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:09:54.560 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:09:54.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:09:54.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:54.830 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:54.831+0000 7f8920049640 -1 Falling back to public interface 2024-02-05T05:09:54.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:55.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:55.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:55.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:09:55.891 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:09:55.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:55.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:09:56.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:57.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:57.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:57.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:09:57.210 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:09:57.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:57.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:09:57.400 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:09:57.403+0000 7f8920049640 -1 osd.0 21 log_to_monitors true 2024-02-05T05:09:57.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:58.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:58.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:58.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:09:58.525 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:09:58.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:58.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:09:58.881 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3374504837,v1:127.0.0.1:6803/3374504837] [v2:127.0.0.1:6804/3374504837,v1:127.0.0.1:6805/3374504837] exists,up 195f6dc1-aa07-4514-ada5-2223daea6e28 2024-02-05T05:09:58.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:09:58.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:09:58.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:09:58.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:09:58.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:09:58.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:09:58.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:09:58.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:09:58.882 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:09:58.882 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:09:58.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:09:58.883 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:09:58.883 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:09:58.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:09:58.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:09:58.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:09:58.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:09:58.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:09:58.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:09:59.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:09:59.309 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:09:59.310 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:09:59.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:09:59.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:59.310 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:09:59.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-02-05T05:09:59.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-02-05T05:09:59.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-02-05T05:09:59.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:59.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:09:59.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-02-05T05:09:59.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-02-05T05:09:59.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672976' 2024-02-05T05:09:59.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:59.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:09:59.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-02-05T05:09:59.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-02-05T05:09:59.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672976 2-64424509457' 2024-02-05T05:09:59.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:59.735 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-02-05T05:09:59.736 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:59.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:09:59.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-02-05T05:09:59.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:59.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-02-05T05:09:59.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-02-05T05:09:59.738 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 107374182402 2024-02-05T05:09:59.738 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:00.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-02-05T05:10:00.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:10:01.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:10:01.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:01.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-02-05T05:10:01.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:01.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-02-05T05:10:01.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:01.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:10:01.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-02-05T05:10:01.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:01.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-02-05T05:10:01.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-02-05T05:10:01.467 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672976 2024-02-05T05:10:01.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:10:01.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-02-05T05:10:01.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:01.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-02-05T05:10:01.769 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:01.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:10:01.770 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-02-05T05:10:01.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:01.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-02-05T05:10:01.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-02-05T05:10:01.772 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509457 2024-02-05T05:10:01.773 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:02.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-02-05T05:10:02.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:10:02.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:02.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:02.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:10:02.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:02.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:02.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:02.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:02.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:02.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:02.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:02.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:10:02.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:02.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:02.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:03.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:10:03.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:10:03.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:10:03.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-02-05T05:10:03.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:10:03.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:10:03.128 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T05:10:03.128 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:10:03.128 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:10:03.128 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:10:03.128 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:10:03.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:10:03.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:10:03.448 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:10:03.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-02-05T05:10:03.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:10:03.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:10:03.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:10:03.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:10:03.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-02-05T05:10:03.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:10:03.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:10:03.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:10:03.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:10:03.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:10:03.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:10:03.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:10:03.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:10:03.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:10:03.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:10:04.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:10:04.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-02-05T05:10:04.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:10:04.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:10:04.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:10:04.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:10:04.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:10:04.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-02-05T05:10:05.234 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:104778fc:::obj2:head# 2024-02-05T05:10:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:10:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:10:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:10:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:10:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:10:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:10:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:10:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:10:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:10:05.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:10:05.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:10:05.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:10:05.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:10:05.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:10:05.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:10:05.773 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:10:05.773 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:10:05.773 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:10:05.775 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:10:05.775 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:05.775 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:10:05.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:10:05.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:10:05.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:10:05.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:10:05.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:10:05.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:10:05.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:10:05.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:10:05.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:10:05.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:10:05.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:10:05.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:10:05.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:10:05.777 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:10:05.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:10:05.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:10:05.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:10:05.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:10:05.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:10:05.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:10:05.812 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:05.815+0000 7f27dbe5c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:05.814 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:05.815+0000 7f27dbe5c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:05.816 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:05.819+0000 7f27dbe5c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:06.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:10:06.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:10:06.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:10:06.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:10:06.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:10:06.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:06.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:10:06.094 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:10:06.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:06.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:10:06.349 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:06.351+0000 7f27dbe5c640 -1 Falling back to public interface 2024-02-05T05:10:06.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:07.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:07.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:07.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:10:07.431 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:10:07.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:07.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:10:07.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:08.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:08.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:08.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:10:08.737 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:10:08.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:08.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:10:08.783 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:08.783+0000 7f27dbe5c640 -1 osd.0 26 log_to_monitors true 2024-02-05T05:10:09.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:10.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:10.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:10.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:10:10.072 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:10:10.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:10.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:10:10.393 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/4003711664,v1:127.0.0.1:6803/4003711664] [v2:127.0.0.1:6804/4003711664,v1:127.0.0.1:6805/4003711664] exists,up 195f6dc1-aa07-4514-ada5-2223daea6e28 2024-02-05T05:10:10.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:10:10.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:10:10.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:10:10.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:10:10.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:10:10.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:10:10.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:10:10.394 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:10:10.394 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:10:10.395 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:10:10.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:10:10.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:10:10.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:10:10.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:10:10.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:10:10.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:10:10.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:10:10.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:10:10.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:10:10.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:10:10.786 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:10:10.786 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:10:10.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:10:10.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:10.786 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:10:10.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-02-05T05:10:10.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-02-05T05:10:10.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-02-05T05:10:10.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:10.942 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:10:11.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-02-05T05:10:11.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-02-05T05:10:11.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672979' 2024-02-05T05:10:11.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:11.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:10:11.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-02-05T05:10:11.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-02-05T05:10:11.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672979 2-64424509460' 2024-02-05T05:10:11.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:11.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-02-05T05:10:11.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:11.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:10:11.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-02-05T05:10:11.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:11.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-02-05T05:10:11.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-02-05T05:10:11.219 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 128849018882 2024-02-05T05:10:11.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:11.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-02-05T05:10:11.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:11.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-02-05T05:10:11.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:11.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:10:11.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:11.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-02-05T05:10:11.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-02-05T05:10:11.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-02-05T05:10:11.512 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672979 2024-02-05T05:10:11.513 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:10:11.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-02-05T05:10:11.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:11.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-02-05T05:10:11.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:11.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:10:11.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-02-05T05:10:11.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:11.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-02-05T05:10:11.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-02-05T05:10:11.805 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509460 2024-02-05T05:10:11.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:12.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-02-05T05:10:12.099 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:10:12.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:12.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:12.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:10:12.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:12.489 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:12.489 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:12.489 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:12.489 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:12.489 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:12.489 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:12.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:10:12.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:12.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:12.781 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:13.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:10:13.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:10:13.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:10:13.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-02-05T05:10:13.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:10:13.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:10:13.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:10:13.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:10:13.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-02-05T05:10:13.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:10:13.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:10:13.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:10:13.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:10:13.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-02-05T05:10:13.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:10:13.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:10:13.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:10:13.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:10:13.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:10:13.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:10:13.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:10:13.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:10:13.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:10:13.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:10:13.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:10:13.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-02-05T05:10:13.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:10:13.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:10:13.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:10:13.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:10:13.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:10:13.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-02-05T05:10:15.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:10:15.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:10:15.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:10:15.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:10:15.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:10:15.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:10:15.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:10:15.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:10:15.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:10:15.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:10:15.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:10:15.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:10:15.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:10:15.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:10:15.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:10:15.569 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:10:15.569 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:10:15.569 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:10:15.570 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:10:15.570 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:15.570 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:10:15.570 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:10:15.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:10:15.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:10:15.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:10:15.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:10:15.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:10:15.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:10:15.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:10:15.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:10:15.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:10:15.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:10:15.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:10:15.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:10:15.572 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:10:15.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:10:15.572 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:10:15.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:10:15.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:10:15.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:10:15.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:10:15.608 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:15.611+0000 7f712598b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:15.611 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:15.611+0000 7f712598b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:15.612 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:15.615+0000 7f712598b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:15.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:10:15.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:10:15.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:10:15.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:10:15.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:10:15.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:15.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:10:15.891 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:10:15.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:15.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:16.150 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:16.151+0000 7f712598b640 -1 Falling back to public interface 2024-02-05T05:10:16.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:17.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:17.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:17.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:10:17.225 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:10:17.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:17.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:17.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:18.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:18.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:18.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:10:18.534 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:10:18.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:18.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:18.645 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:18.647+0000 7f712598b640 -1 osd.1 31 log_to_monitors true 2024-02-05T05:10:18.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:19.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:19.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:19.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:10:19.851 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:10:19.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:19.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:20.159 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/1878846835,v1:127.0.0.1:6811/1878846835] [v2:127.0.0.1:6812/1878846835,v1:127.0.0.1:6813/1878846835] exists,up 5838c5b9-eeba-4a7f-a55b-62a337768543 2024-02-05T05:10:20.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:10:20.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:10:20.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:10:20.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:10:20.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:10:20.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:10:20.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:10:20.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:10:20.161 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:10:20.161 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:10:20.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:10:20.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:10:20.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:10:20.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:10:20.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:10:20.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:10:20.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:10:20.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:10:20.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:10:20.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:10:20.553 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:10:20.553 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:10:20.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:10:20.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:20.553 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:10:20.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-02-05T05:10:20.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-02-05T05:10:20.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885' 2024-02-05T05:10:20.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:20.712 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:10:20.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-02-05T05:10:20.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-02-05T05:10:20.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362' 2024-02-05T05:10:20.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:20.854 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:10:20.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-02-05T05:10:20.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-02-05T05:10:20.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362 2-64424509463' 2024-02-05T05:10:20.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:20.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018885 2024-02-05T05:10:20.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:20.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:10:20.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018885 2024-02-05T05:10:20.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:20.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-02-05T05:10:20.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-02-05T05:10:20.989 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 128849018885 2024-02-05T05:10:20.989 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:21.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018884 -lt 128849018885 2024-02-05T05:10:21.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:10:22.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:10:22.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:22.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-02-05T05:10:22.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:22.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-02-05T05:10:22.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:22.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:10:22.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-02-05T05:10:22.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:22.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-02-05T05:10:22.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-02-05T05:10:22.596 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 150323855362 2024-02-05T05:10:22.597 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:10:22.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-02-05T05:10:22.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:22.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-02-05T05:10:22.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:22.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:10:22.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-02-05T05:10:22.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:22.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-02-05T05:10:22.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-02-05T05:10:22.914 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509463 2024-02-05T05:10:22.914 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:23.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-02-05T05:10:23.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:10:23.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:23.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:23.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:10:23.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:23.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:23.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:23.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:23.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:23.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:23.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:23.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:10:23.905 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:23.905 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:23.906 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:24.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:10:24.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:10:24.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:10:24.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-02-05T05:10:24.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-02-05T05:10:24.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-02-05T05:10:24.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-02-05T05:10:24.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:10:24.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-02-05T05:10:24.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-02-05T05:10:24.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:10:24.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T05:10:24.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:10:24.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-02-05T05:10:24.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-02-05T05:10:24.892 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-02-05T05:10:24.892 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:10:24.892 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:10:24.893 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:10:24.893 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:10:25.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-02-05T05:09:41.916124+0000 2024-02-05T05:10:25.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-02-05T05:10:25.329 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T05:10:25.329 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": true, 2024-02-05T05:10:25.329 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T05:10:25.329 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-22T05:08:45.336041+0000" 2024-02-05T05:10:25.330 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T05:10:25.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-02-05T05:09:41.916124+0000 2024-02-05T05:10:25.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T05:10:25.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:09:41.916124+0000 2024-02-05T05:10:25.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:10:25.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:10:25.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:10:25.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:10:25.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:10:25.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:10:25.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:10:25.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:10:25.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:09:41.916124+0000 '>' 2024-02-05T05:09:41.916124+0000 2024-02-05T05:10:25.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:10:26.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:10:26.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:10:26.648 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:10:26.648 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:10:26.648 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:10:26.649 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:10:26.649 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:10:27.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:09:41.916124+0000 '>' 2024-02-05T05:09:41.916124+0000 2024-02-05T05:10:27.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:10:28.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:10:28.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:10:28.005 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:10:28.005 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:10:28.005 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:10:28.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:10:28.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:10:28.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:09:41.916124+0000 '>' 2024-02-05T05:09:41.916124+0000 2024-02-05T05:10:28.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:10:29.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:10:29.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:10:29.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:10:29.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:10:29.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:10:29.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:10:29.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:10:29.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:10:25.948875+0000 '>' 2024-02-05T05:09:41.916124+0000 2024-02-05T05:10:29.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:10:29.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-02-05T05:10:29.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:10:29.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:10:29.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:10:29.679 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:10:29.680 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:10:29.680 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:10:29.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:10:29.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:10:29.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:10:29.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:10:29.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:10:29.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:10:29.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:10:29.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:10:29.786 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:10:30.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:10:30.096 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:10:30.096 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:10:30.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:10:30.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:30.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:10:30.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018888 2024-02-05T05:10:30.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018888 2024-02-05T05:10:30.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888' 2024-02-05T05:10:30.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:30.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:10:30.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-02-05T05:10:30.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-02-05T05:10:30.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888 1-150323855365' 2024-02-05T05:10:30.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:30.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:10:30.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509466 2024-02-05T05:10:30.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509466 2024-02-05T05:10:30.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888 1-150323855365 2-64424509466' 2024-02-05T05:10:30.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:30.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018888 2024-02-05T05:10:30.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:30.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:10:30.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018888 2024-02-05T05:10:30.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:30.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018888 2024-02-05T05:10:30.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018888' 2024-02-05T05:10:30.542 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 128849018888 2024-02-05T05:10:30.542 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:30.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018888 2024-02-05T05:10:30.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:10:31.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:10:31.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:32.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018888 -lt 128849018888 2024-02-05T05:10:32.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:32.167 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:32.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855365 2024-02-05T05:10:32.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:10:32.170 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855365 2024-02-05T05:10:32.170 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:32.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-02-05T05:10:32.171 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 150323855365 2024-02-05T05:10:32.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-02-05T05:10:32.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:10:32.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-02-05T05:10:32.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:32.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509466 2024-02-05T05:10:32.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:32.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:10:32.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509466 2024-02-05T05:10:32.476 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:32.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509466 2024-02-05T05:10:32.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-02-05T05:10:32.477 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509466 2024-02-05T05:10:32.477 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:32.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509466 -lt 64424509466 2024-02-05T05:10:32.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:10:32.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:32.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:33.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:10:33.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:33.163 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:33.163 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:33.163 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:33.163 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:33.163 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:33.163 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:33.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:10:33.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:33.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:33.501 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:33.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:10:33.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:10:33.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:10:33.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-02-05T05:10:33.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:10:33.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:10:34.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:10:34.147 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:10:34.147 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:10:34.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:10:34.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:34.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:10:34.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018890 2024-02-05T05:10:34.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018890 2024-02-05T05:10:34.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890' 2024-02-05T05:10:34.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:34.308 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:10:34.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855367 2024-02-05T05:10:34.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855367 2024-02-05T05:10:34.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855367' 2024-02-05T05:10:34.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:34.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:10:34.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509468 2024-02-05T05:10:34.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509468 2024-02-05T05:10:34.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855367 2-64424509468' 2024-02-05T05:10:34.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:34.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018890 2024-02-05T05:10:34.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:34.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:10:34.586 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018890 2024-02-05T05:10:34.586 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:34.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018890 2024-02-05T05:10:34.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018890' 2024-02-05T05:10:34.587 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 128849018890 2024-02-05T05:10:34.588 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:34.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018888 -lt 128849018890 2024-02-05T05:10:34.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:10:35.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:10:35.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:36.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018890 -lt 128849018890 2024-02-05T05:10:36.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:36.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855367 2024-02-05T05:10:36.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:36.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:10:36.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855367 2024-02-05T05:10:36.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:36.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855367 2024-02-05T05:10:36.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855367' 2024-02-05T05:10:36.181 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 150323855367 2024-02-05T05:10:36.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:10:36.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855367 -lt 150323855367 2024-02-05T05:10:36.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:36.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509468 2024-02-05T05:10:36.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:36.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:10:36.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509468 2024-02-05T05:10:36.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:36.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509468 2024-02-05T05:10:36.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-02-05T05:10:36.502 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509468 2024-02-05T05:10:36.503 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:36.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509468 -lt 64424509468 2024-02-05T05:10:36.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-02-05T05:10:36.861 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T05:10:36.861 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-02-05T05:10:36.861 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-02-05T05:10:36.861 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-02-05T05:10:36.861 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T05:10:36.861 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.943+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T05:10:36.861 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.943+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-02-05T05:10:36.861 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.943+0000 7f7109915640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T05:10:36.861 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.943+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-02-05T05:10:36.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-02-05T05:10:36.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-02-05T05:10:37.165 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T05:10:37.165 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-02-05T05:10:25.949080+0000 21'10 36:94 [1,0] 1 [1,0] 1 21'10 2024-02-05T05:10:25.948875+0000 21'10 2024-02-05T05:10:25.948875+0000 0 1 periodic scrub scheduled @ 2024-02-06T05:10:25.948875+0000 10 0 2024-02-05T05:10:37.165 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T05:10:37.165 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T05:10:37.166 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:10:37.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-02-05T05:10:37.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-02-05T05:10:37.465 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:10:37.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-02-05T05:10:37.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:10:37.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-02-05T05:10:37.482 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-02-05T05:10:37.483 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:10:37.483 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T05:10:37.483 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-02-05T05:10:37.483 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:10:37.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:10:37.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-02-05T05:10:37.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:10:38.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-02-05T05:10:38.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:10:38.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:10:38.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:10:38.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:10:38.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-02-05T05:10:38.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:10:38.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:10:38.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:10:38.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:10:38.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:10:38.121 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:10:38.121 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:10:38.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:10:38.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:10:38.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:10:38.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:10:38.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-02-05T05:10:38.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:10:38.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:10:38.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:10:38.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:10:38.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:10:38.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-02-05T05:10:39.800 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:10:39.800 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:10:40.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:10:40.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:10:40.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:10:40.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:10:40.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:10:40.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:10:40.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:10:40.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:10:40.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:10:40.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:10:40.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:10:40.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:10:40.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:10:40.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:10:40.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:10:40.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:10:40.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:10:40.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:10:40.337 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:10:40.337 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:40.337 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:10:40.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:10:40.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:10:40.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:10:40.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:10:40.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:10:40.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:10:40.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:10:40.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:10:40.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:10:40.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:10:40.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:10:40.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:10:40.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:10:40.339 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:10:40.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:10:40.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:10:40.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:10:40.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:10:40.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:10:40.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:10:40.377 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:40.378+0000 7f6221c10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:40.379 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:40.382+0000 7f6221c10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:40.381 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:40.382+0000 7f6221c10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:40.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:10:40.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:10:40.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:10:40.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:10:40.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:10:40.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:40.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:10:40.651 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:10:40.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:40.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:10:40.929 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:40.930+0000 7f6221c10640 -1 Falling back to public interface 2024-02-05T05:10:40.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:41.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:41.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:41.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:10:41.982 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:10:41.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:41.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:10:42.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:43.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:43.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:43.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:10:43.289 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:10:43.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:43.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:10:43.478 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:43.478+0000 7f6221c10640 -1 osd.0 36 log_to_monitors true 2024-02-05T05:10:43.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:44.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:44.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:44.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:10:44.605 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:10:44.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:44.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:10:44.971 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/2436581197,v1:127.0.0.1:6803/2436581197] [v2:127.0.0.1:6804/2436581197,v1:127.0.0.1:6805/2436581197] exists,up 195f6dc1-aa07-4514-ada5-2223daea6e28 2024-02-05T05:10:44.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:10:44.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:10:44.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:10:44.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:10:44.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:10:44.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:10:44.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:10:44.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:10:44.972 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:10:44.972 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:10:44.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:10:44.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:10:44.973 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:10:45.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:10:45.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:10:45.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:10:45.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:10:45.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:10:45.075 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:10:45.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:10:45.396 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:10:45.396 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:10:45.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:10:45.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:45.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:10:45.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-02-05T05:10:45.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-02-05T05:10:45.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842' 2024-02-05T05:10:45.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:45.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:10:45.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855370 2024-02-05T05:10:45.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855370 2024-02-05T05:10:45.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370' 2024-02-05T05:10:45.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:45.687 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:10:45.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509471 2024-02-05T05:10:45.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509471 2024-02-05T05:10:45.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370 2-64424509471' 2024-02-05T05:10:45.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:45.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691842 2024-02-05T05:10:45.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:45.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:10:45.833 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691842 2024-02-05T05:10:45.833 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-02-05T05:10:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-02-05T05:10:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 171798691842 2024-02-05T05:10:45.835 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:46.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 171798691842 2024-02-05T05:10:46.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:10:47.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:10:47.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:47.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-02-05T05:10:47.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:47.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855370 2024-02-05T05:10:47.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:47.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:10:47.477 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855370 2024-02-05T05:10:47.477 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:47.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855370 2024-02-05T05:10:47.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855370' 2024-02-05T05:10:47.479 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 150323855370 2024-02-05T05:10:47.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:10:47.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855370 2024-02-05T05:10:47.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:47.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:47.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509471 2024-02-05T05:10:47.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:10:47.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509471 2024-02-05T05:10:47.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:47.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509471 2024-02-05T05:10:47.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-02-05T05:10:47.807 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509471 2024-02-05T05:10:47.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:48.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509471 -lt 64424509471 2024-02-05T05:10:48.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:10:48.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:48.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:48.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:10:48.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:48.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:48.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:48.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:48.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:48.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:48.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:48.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:10:48.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:48.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:48.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:49.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:10:49.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:10:49.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:10:49.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-02-05T05:10:49.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:10:49.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-02-05T05:10:49.226 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T05:10:49.226 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-02-05T05:10:49.226 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-02-05T05:10:49.226 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-02-05T05:10:49.226 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T05:10:49.226 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.943+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T05:10:49.226 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.943+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-02-05T05:10:49.226 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.943+0000 7f7109915640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T05:10:49.227 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.943+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-02-05T05:10:49.227 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-02-05T05:10:49.227 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:10:49.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:10:49.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:10:49.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:10:49.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-02-05T05:10:49.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:10:49.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:10:49.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:10:49.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:10:49.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-02-05T05:10:49.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:10:49.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:10:49.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:10:49.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:10:49.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:10:49.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:10:49.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:10:49.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:10:49.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:10:49.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:10:49.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:10:49.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-02-05T05:10:49.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:10:49.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:10:49.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:10:49.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:10:49.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:10:49.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-02-05T05:10:50.991 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-02-05T05:10:50.991 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:104778fc:::obj2:head# 2024-02-05T05:10:51.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:10:51.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:10:51.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:10:51.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:10:51.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:10:51.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:10:51.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:10:51.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:10:51.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:10:51.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:10:51.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:10:51.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:10:51.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:10:51.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:10:51.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:10:51.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:10:51.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:10:51.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:10:51.536 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:10:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:10:51.537 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:10:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:10:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:10:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:10:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:10:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:10:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:10:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:10:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:10:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:10:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:10:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:10:51.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:10:51.538 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:10:51.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:10:51.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:10:51.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:10:51.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:10:51.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:10:51.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:10:51.575 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:51.574+0000 7fe2681d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:51.577 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:51.578+0000 7fe2681d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:51.579 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:51.578+0000 7fe2681d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:51.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:10:51.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:10:51.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:10:51.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:10:51.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:10:51.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:51.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:10:51.854 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:10:51.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:51.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:52.122 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:52.122+0000 7fe2681d7640 -1 Falling back to public interface 2024-02-05T05:10:52.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:53.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:53.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:53.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:10:53.167 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:10:53.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:53.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:53.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:54.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:54.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:54.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:10:54.502 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:10:54.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:54.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:54.612 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:10:54.614+0000 7fe2681d7640 -1 osd.1 41 log_to_monitors true 2024-02-05T05:10:54.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:55.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:55.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:55.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:10:55.821 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:10:55.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:55.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:56.154 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/3048549047,v1:127.0.0.1:6811/3048549047] [v2:127.0.0.1:6812/3048549047,v1:127.0.0.1:6813/3048549047] exists,up 5838c5b9-eeba-4a7f-a55b-62a337768543 2024-02-05T05:10:56.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:10:56.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:10:56.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:10:56.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:10:56.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:10:56.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:10:56.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:10:56.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:10:56.155 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:10:56.156 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:10:56.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:10:56.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:10:56.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:10:56.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:10:56.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:10:56.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:10:56.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:10:56.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:10:56.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:10:56.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:10:56.572 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:10:56.572 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:10:56.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:10:56.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:56.573 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:10:56.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-02-05T05:10:56.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-02-05T05:10:56.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845' 2024-02-05T05:10:56.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:56.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:10:56.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528322 2024-02-05T05:10:56.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528322 2024-02-05T05:10:56.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322' 2024-02-05T05:10:56.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:56.865 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:10:56.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509474 2024-02-05T05:10:56.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509474 2024-02-05T05:10:56.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322 2-64424509474' 2024-02-05T05:10:56.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:56.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691845 2024-02-05T05:10:56.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:56.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:10:56.997 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691845 2024-02-05T05:10:56.997 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:56.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-02-05T05:10:56.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-02-05T05:10:56.998 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 171798691845 2024-02-05T05:10:56.998 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:57.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691844 -lt 171798691845 2024-02-05T05:10:57.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:10:58.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:10:58.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:58.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-02-05T05:10:58.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:58.649 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528322 2024-02-05T05:10:58.649 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:58.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:10:58.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528322 2024-02-05T05:10:58.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:58.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528322 2024-02-05T05:10:58.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2024-02-05T05:10:58.654 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 193273528322 2024-02-05T05:10:58.654 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:10:58.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528322 -lt 193273528322 2024-02-05T05:10:58.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:58.960 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509474 2024-02-05T05:10:58.960 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:58.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:10:58.963 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509474 2024-02-05T05:10:58.963 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:58.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509474 2024-02-05T05:10:58.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-02-05T05:10:58.964 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509474 2024-02-05T05:10:58.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:59.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509474 -lt 64424509474 2024-02-05T05:10:59.285 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:10:59.285 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:59.286 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:59.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:10:59.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:59.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:59.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:59.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:59.681 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:59.681 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:59.681 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:11:00.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:11:00.013 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:11:00.013 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:11:00.013 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:11:00.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:11:00.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:11:00.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:11:00.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-02-05T05:11:00.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-02-05T05:11:00.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-02-05T05:11:00.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:11:00.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:11:00.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:11:00.359 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:11:00.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:10:25.948875+0000 2024-02-05T05:11:00.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-02-05T05:11:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 1.0 on osd.1 to repair 2024-02-05T05:11:00.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-02-05T05:10:25.948875+0000 2024-02-05T05:11:00.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T05:11:00.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:10:25.948875+0000 2024-02-05T05:11:00.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:11:00.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:11:00.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:11:00.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:11:00.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:11:00.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:11:00.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:11:00.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:11:01.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:10:25.948875+0000 '>' 2024-02-05T05:10:25.948875+0000 2024-02-05T05:11:01.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:11:02.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:11:02.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:11:02.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:11:02.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:11:02.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:11:02.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:11:02.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:11:02.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:10:25.948875+0000 '>' 2024-02-05T05:10:25.948875+0000 2024-02-05T05:11:02.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:11:03.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:11:03.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:11:03.597 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:11:03.597 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:11:03.597 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:11:03.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:11:03.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:11:03.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:10:25.948875+0000 '>' 2024-02-05T05:10:25.948875+0000 2024-02-05T05:11:03.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:11:04.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:11:04.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:11:04.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:11:04.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:11:04.904 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:11:04.904 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:11:04.904 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:11:05.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:10:25.948875+0000 '>' 2024-02-05T05:10:25.948875+0000 2024-02-05T05:11:05.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:11:06.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:11:06.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:11:06.201 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:11:06.201 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:11:06.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:11:06.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:11:06.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:11:06.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:11:01.725382+0000 '>' 2024-02-05T05:10:25.948875+0000 2024-02-05T05:11:06.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:11:06.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-02-05T05:11:08.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-02-05T05:11:08.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:11:08.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:11:08.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:11:08.825 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:11:08.825 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:11:08.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:11:08.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:11:08.826 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:11:08.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691849 2024-02-05T05:11:08.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691849 2024-02-05T05:11:08.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849' 2024-02-05T05:11:08.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:11:08.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:11:09.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528325 2024-02-05T05:11:09.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528325 2024-02-05T05:11:09.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849 1-193273528325' 2024-02-05T05:11:09.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:11:09.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:11:09.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509478 2024-02-05T05:11:09.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509478 2024-02-05T05:11:09.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849 1-193273528325 2-64424509478' 2024-02-05T05:11:09.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:11:09.270 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691849 2024-02-05T05:11:09.270 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:11:09.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:11:09.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691849 2024-02-05T05:11:09.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:11:09.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691849 2024-02-05T05:11:09.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691849' 2024-02-05T05:11:09.273 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 171798691849 2024-02-05T05:11:09.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:11:09.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691849 -lt 171798691849 2024-02-05T05:11:09.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:11:09.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528325 2024-02-05T05:11:09.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:11:09.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:11:09.566 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528325 2024-02-05T05:11:09.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:11:09.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528325 2024-02-05T05:11:09.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528325' 2024-02-05T05:11:09.568 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 193273528325 2024-02-05T05:11:09.569 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:11:09.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528325 -lt 193273528325 2024-02-05T05:11:09.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:11:09.904 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509478 2024-02-05T05:11:09.904 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:11:09.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:11:09.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509478 2024-02-05T05:11:09.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:11:09.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509478 2024-02-05T05:11:09.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2024-02-05T05:11:09.909 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509478 2024-02-05T05:11:09.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:11:10.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509477 -lt 64424509478 2024-02-05T05:11:10.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:11:11.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:11:11.233 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:11:11.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509478 -lt 64424509478 2024-02-05T05:11:11.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-02-05T05:11:11.840 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T05:11:11.840 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-02-05T05:11:01.725654+0000 21'10 46:130 [1,0] 1 [1,0] 1 21'10 2024-02-05T05:11:01.725382+0000 21'10 2024-02-05T05:11:01.725382+0000 0 1 periodic scrub scheduled @ 2024-02-06T05:11:01.725382+0000 9 0 2024-02-05T05:11:11.840 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T05:11:11.840 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T05:11:11.841 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:11:11.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-02-05T05:11:11.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-02-05T05:11:12.133 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:11:12.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-02-05T05:11:12.153 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T05:11:12.153 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-02-05T05:11:12.153 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-02-05T05:11:12.153 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-02-05T05:11:12.153 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.683+0000 7f7109915640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T05:11:12.154 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.943+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T05:11:12.154 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.943+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-02-05T05:11:12.154 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.943+0000 7f7109915640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T05:11:12.154 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:10:25.943+0000 7f7109915640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-02-05T05:11:12.154 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:11:01.722+0000 7fe24c161640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR planned AUTO_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-02-05T05:11:12.154 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:11:01.722+0000 7fe24c161640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-02-05T05:11:12.154 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:11:01.722+0000 7fe24c161640 15 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-02-05T05:11:12.154 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:11:01.722+0000 7fe24c161640 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-02-05T05:11:12.154 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:11:01.722+0000 7fe24c161640 19 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T05:11:12.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:11:12.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:11:12.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:11:12.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:11:12.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:11:12.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:11:12.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:11:12.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:11:12.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:11:12.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:11:12.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:11:12.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:11:12.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:11:12.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:11:12.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:11:12.269 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:11:12.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:11:12.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:11:12.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:11:12.270 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:11:12.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:11:12.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:11:12.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:11:12.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:11:12.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:11:12.290 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:11:12.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:11:12.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:11:12.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:11:12.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:11:12.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:11:12.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:11:12.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:11:12.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:11:12.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:11:12.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:11:12.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:11:12.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:11:12.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:11:12.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:11:12.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:11:12.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:11:12.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:11:12.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:11:12.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:11:12.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:11:12.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:11:12.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:11:12.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:11:12.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:11:12.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:11:12.301 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:11:12.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:11:12.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:11:12.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:11:12.301 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:11:12.301 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:11:12.301 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:11:12.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:11:12.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:11:12.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:11:12.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:11:12.306 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:11:12.306 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:11:12.306 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:11:12.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T05:11:12.306 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:11:12.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:11:12.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:11:12.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:11:12.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-02-05T05:11:12.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-02-05T05:11:12.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-02-05T05:11:12.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-02-05T05:11:12.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:11:12.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:11:12.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:11:12.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:11:12.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:11:12.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:11:12.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:11:12.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:11:12.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:11:12.699 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:11:12.699 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:11:12.699 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:11:12.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:11:12.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:11:12.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:11:12.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:11:12.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:11:12.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:11:12.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:11:12.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:11:12.912 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:11:12.913 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:11:12.913 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:11:12.913 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:11:12.913 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:11:12.913 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:11:12.913 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:11:12.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:11:12.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T05:11:13.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:11:13.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:11:13.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:11:13.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:11:13.031 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:11:13.031 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:11:13.032 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:11:13.032 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:11:13.032 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:11:13.032 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:11:13.032 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:11:13.032 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:11:13.032 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:11:13.032 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T05:11:13.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-02-05T05:11:13.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:11:13.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:11:13.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:11:13.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:11:13.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:11:13.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:11:14.018 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:11:14.018 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:11:14.019 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:11:14.019 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:11:14.019 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:11:14.019 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:11:14.019 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:11:14.020 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:11:14.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:11:14.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:11:14.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-02-05T05:11:14.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-02-05T05:11:14.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:11:14.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:11:14.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:11:14.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:11:14.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:11:14.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:11:14.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:11:14.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:11:14.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:11:14.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:11:14.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:11:14.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:11:14.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:11:14.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:11:14.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:11:14.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:11:14.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:11:14.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:11:14.058 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:11:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:11:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:11:14.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:11:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:11:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:11:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:11:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:11:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:11:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:11:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:11:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:11:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:11:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:11:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:11:14.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:11:14.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:11:14.060 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:11:14.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a1812611-9de1-418e-b494-25a83141fa20 2024-02-05T05:11:14.060 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 a1812611-9de1-418e-b494-25a83141fa20 2024-02-05T05:11:14.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a1812611-9de1-418e-b494-25a83141fa20' 2024-02-05T05:11:14.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:11:14.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDybcBl9iLuBBAAyZKT1aLz9FuZKNqHRY8muQ== 2024-02-05T05:11:14.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDybcBl9iLuBBAAyZKT1aLz9FuZKNqHRY8muQ=="}' 2024-02-05T05:11:14.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a1812611-9de1-418e-b494-25a83141fa20 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:11:14.321 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:11:14.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:11:14.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDybcBl9iLuBBAAyZKT1aLz9FuZKNqHRY8muQ== --osd-uuid a1812611-9de1-418e-b494-25a83141fa20 2024-02-05T05:11:14.377 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:14.382+0000 7f913f7b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:14.379 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:14.386+0000 7f913f7b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:14.380 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:14.386+0000 7f913f7b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:14.381 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:14.386+0000 7f913f7b6640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:11:14.381 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:14.386+0000 7f913f7b6640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:11:14.381 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:14.386+0000 7f913f7b6640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:11:14.464 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:14.470+0000 7f913f7b6640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:11:20.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:11:20.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:11:20.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:11:20.705 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:11:20.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:11:21.090 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:11:21.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:11:21.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:11:21.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:11:21.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:11:21.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:11:21.125 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:21.130+0000 7fe09e655640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:21.156 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:21.162+0000 7fe09e655640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:21.157 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:21.162+0000 7fe09e655640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:21.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:11:21.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:11:21.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:11:21.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:11:21.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:11:21.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:21.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:11:21.433 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:11:21.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:21.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:21.705 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:21.710+0000 7fe09e655640 -1 Falling back to public interface 2024-02-05T05:11:21.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:22.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:22.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:22.731 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:11:22.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:11:22.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:22.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:23.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:24.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:24.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:24.040 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:11:24.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:11:24.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:24.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:24.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:24.369 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:24.374+0000 7fe09e655640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:11:25.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:25.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:25.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:11:25.366 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:11:25.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:25.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:25.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:11:26.676 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:11:26.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:26.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:26.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:27.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:27.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:27.988 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:11:27.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:11:27.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:27.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:28.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:29.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:29.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:29.478 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:11:29.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:11:29.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:29.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:29.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:30.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:30.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:30.800 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:11:30.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:11:30.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:30.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:31.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:32.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:32.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:32.115 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:11:32.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:11:32.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:32.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:32.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:33.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:33.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:33.490 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:11:33.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:11:33.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:33.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:33.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:34.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:34.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:34.806 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:11:34.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:11:34.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:34.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:35.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:36.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:36.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:36.110 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:11:36.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:11:36.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:36.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:36.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:37.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:37.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:37.416 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:11:37.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:11:37.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:37.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:37.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:38.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:38.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:38.738 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:11:38.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:11:38.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:38.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:39.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:40.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:40.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:40.424 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:11:40.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:11:40.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:40.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:41.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:42.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:42.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:42.257 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:11:42.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:11:42.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:42.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:42.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:43.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:43.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:43.755 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:11:43.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:11:43.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:43.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:44.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:45.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:45.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:45.660 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:11:45.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:11:45.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:45.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:46.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:47.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:47.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:47.156 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:11:47.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:11:47.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:47.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:47.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:48.728 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:11:48.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:11:48.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:48.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:49.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:50.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:50.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:50.119 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:11:50.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:11:50.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:50.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:50.439 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2979689996,v1:127.0.0.1:6803/2979689996] [v2:127.0.0.1:6804/2979689996,v1:127.0.0.1:6805/2979689996] exists,up a1812611-9de1-418e-b494-25a83141fa20 2024-02-05T05:11:50.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:11:50.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:11:50.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:11:50.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-02-05T05:11:50.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:11:50.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:11:50.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:11:50.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:11:50.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:11:50.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:11:50.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:11:50.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:11:50.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:11:50.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:11:50.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:11:50.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:11:50.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:11:50.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:11:50.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:11:50.442 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:11:50.442 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:11:50.442 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:11:50.444 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:11:50.444 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:11:50.444 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:11:50.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:11:50.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:11:50.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:11:50.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:11:50.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:11:50.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:11:50.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:11:50.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:11:50.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:11:50.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:11:50.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:11:50.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:11:50.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:11:50.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:11:50.446 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:11:50.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=368b3717-439e-43b4-bd6f-31bc16e05190 2024-02-05T05:11:50.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 368b3717-439e-43b4-bd6f-31bc16e05190' 2024-02-05T05:11:50.447 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 368b3717-439e-43b4-bd6f-31bc16e05190 2024-02-05T05:11:50.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:11:50.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAWbsBl/BdiHBAAIB+/4kNFTRBhR/MxSV69Cw== 2024-02-05T05:11:50.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAWbsBl/BdiHBAAIB+/4kNFTRBhR/MxSV69Cw=="}' 2024-02-05T05:11:50.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 368b3717-439e-43b4-bd6f-31bc16e05190 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:11:50.919 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:11:50.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:11:50.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAWbsBl/BdiHBAAIB+/4kNFTRBhR/MxSV69Cw== --osd-uuid 368b3717-439e-43b4-bd6f-31bc16e05190 2024-02-05T05:11:50.980 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:50.985+0000 7f02c5c9f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:51.004 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:51.009+0000 7f02c5c9f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:51.005 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:51.009+0000 7f02c5c9f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:51.006 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:51.009+0000 7f02c5c9f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:11:51.006 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:51.009+0000 7f02c5c9f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:11:51.006 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:51.009+0000 7f02c5c9f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:11:51.092 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:51.093+0000 7f02c5c9f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:11:57.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:11:57.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:11:57.932 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:11:57.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:11:57.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:11:58.309 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:11:58.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:11:58.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:11:58.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:11:58.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:11:58.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:11:58.343 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:58.349+0000 7f63fa06b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:58.367 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:58.373+0000 7f63fa06b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:58.369 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:58.373+0000 7f63fa06b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:58.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:11:58.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:11:58.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:11:58.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:11:58.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:11:58.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:58.621 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:11:58.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:11:58.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:58.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:58.917 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:11:58.921+0000 7f63fa06b640 -1 Falling back to public interface 2024-02-05T05:11:58.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:59.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:59.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:59.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:11:59.976 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:11:59.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:59.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:00.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:01.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:01.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:01.276 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:12:01.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:12:01.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:01.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:01.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:01.712 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:12:01.717+0000 7f63fa06b640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:12:02.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:02.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:02.586 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:12:02.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:12:02.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:02.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:03.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:04.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:04.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:04.019 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:12:04.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:12:04.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:04.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:04.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:05.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:05.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:05.340 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:12:05.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:12:05.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:05.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:05.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:06.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:06.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:06.805 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:12:06.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:12:06.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:06.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:07.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:08.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:08.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:08.124 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:12:08.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:12:08.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:08.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:08.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:09.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:09.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:09.443 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:12:09.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:12:09.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:09.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:09.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:10.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:10.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:10.816 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:12:10.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:12:10.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:10.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:11.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:12.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:12.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:12.170 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:12:12.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:12:12.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:12.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:12.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:13.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:13.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:13.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:12:13.489 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:12:13.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:13.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:13.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:14.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:14.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:14.849 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:12:14.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:12:14.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:14.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:15.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:16.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:16.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:16.164 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:12:16.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:12:16.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:16.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:16.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:17.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:17.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:17.597 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:12:17.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:12:17.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:17.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:18.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:19.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:19.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:19.395 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:12:19.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:12:19.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:19.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:20.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:21.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:21.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:21.295 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:12:21.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:12:21.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:21.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:22.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:23.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:23.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:23.107 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:12:23.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:12:23.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:23.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:23.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:24.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:24.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:24.536 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:12:24.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:12:24.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:24.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:25.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:26.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:26.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:26.202 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:12:26.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:12:26.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:26.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:26.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:27.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:27.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:27.935 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:12:27.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:12:27.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:27.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:28.258 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2579423163,v1:127.0.0.1:6811/2579423163] [v2:127.0.0.1:6812/2579423163,v1:127.0.0.1:6813/2579423163] exists,up 368b3717-439e-43b4-bd6f-31bc16e05190 2024-02-05T05:12:28.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:12:28.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:12:28.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:12:28.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-02-05T05:12:28.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:12:28.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:12:28.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:12:28.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:12:28.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:12:28.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:12:28.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:12:28.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:12:28.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:12:28.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:12:28.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:12:28.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:12:28.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:12:28.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:12:28.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:12:28.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:12:28.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:12:28.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:12:28.262 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:12:28.262 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:12:28.262 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:12:28.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:12:28.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:12:28.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:12:28.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:12:28.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:12:28.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:12:28.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:12:28.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:12:28.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:12:28.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:12:28.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:12:28.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:12:28.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:12:28.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:12:28.265 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:12:28.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b647939a-edd3-4f8a-9297-87925944112a 2024-02-05T05:12:28.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b647939a-edd3-4f8a-9297-87925944112a' 2024-02-05T05:12:28.265 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 b647939a-edd3-4f8a-9297-87925944112a 2024-02-05T05:12:28.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:12:28.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8bsBlFYOGERAAW3iJ18ag0d5pNGJQmT19sA== 2024-02-05T05:12:28.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8bsBlFYOGERAAW3iJ18ag0d5pNGJQmT19sA=="}' 2024-02-05T05:12:28.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b647939a-edd3-4f8a-9297-87925944112a -i td/osd-scrub-repair/2/new.json 2024-02-05T05:12:28.666 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:12:28.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:12:28.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA8bsBlFYOGERAAW3iJ18ag0d5pNGJQmT19sA== --osd-uuid b647939a-edd3-4f8a-9297-87925944112a 2024-02-05T05:12:28.724 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:12:28.729+0000 7fb986286640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:28.768 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:12:28.773+0000 7fb986286640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:28.770 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:12:28.773+0000 7fb986286640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:28.770 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:12:28.777+0000 7fb986286640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:12:28.770 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:12:28.777+0000 7fb986286640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:12:28.770 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:12:28.777+0000 7fb986286640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:12:28.852 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:12:28.857+0000 7fb986286640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:12:35.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:12:35.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:12:35.956 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T05:12:35.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:12:35.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:12:36.330 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T05:12:36.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:12:36.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:12:36.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:12:36.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:12:36.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:12:36.364 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:12:36.369+0000 7f336f146640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:36.401 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:12:36.405+0000 7f336f146640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:36.402 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:12:36.409+0000 7f336f146640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:36.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:12:36.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:12:36.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:12:36.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:12:36.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:12:36.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:36.619 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:12:36.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:12:36.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:36.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:36.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:36.984 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:12:36.989+0000 7f336f146640 -1 Falling back to public interface 2024-02-05T05:12:37.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:37.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:37.947 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:12:37.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:12:37.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:37.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:38.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:39.246 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:12:39.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:12:39.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:39.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:39.394 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:12:39.401+0000 7f336f146640 -1 osd.2 0 log_to_monitors true 2024-02-05T05:12:39.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:40.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:40.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:40.651 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:12:40.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:12:40.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:40.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:40.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:41.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:41.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:41.968 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:12:41.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:12:41.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:41.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:42.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:43.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:43.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:43.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:12:43.342 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:12:43.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:43.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:43.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:44.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:44.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:44.683 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:12:44.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:12:44.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:44.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:45.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:46.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:46.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:46.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:12:46.007 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:12:46.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:46.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:46.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:47.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:47.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:47.331 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:12:47.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:12:47.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:47.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:47.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:48.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:48.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:48.641 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:12:48.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:12:48.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:48.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:48.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:49.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:49.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:49.950 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:12:49.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:12:49.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:49.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:50.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:51.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:51.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:51.331 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:12:51.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:12:51.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:51.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:51.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:52.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:52.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:52.654 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:12:52.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:12:52.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:52.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:52.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:53.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:53.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:53.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:12:53.969 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:12:53.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:53.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:54.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:55.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:55.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:55.628 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:12:55.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:12:55.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:55.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:56.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:57.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:57.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:57.426 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:12:57.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:12:57.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:57.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:57.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:58.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:58.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:58.843 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:12:58.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:12:58.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:58.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:59.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:00.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:00.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:00.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:13:00.378 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:13:00.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:00.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:13:00.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:01.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:01.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:01.943 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:13:01.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:13:01.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:01.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:13:02.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:03.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:03.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:03.367 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:13:03.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:13:03.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:03.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:13:03.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:04.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:04.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:04.922 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:13:04.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:13:04.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:04.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:13:05.325 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1529974415,v1:127.0.0.1:6819/1529974415] [v2:127.0.0.1:6820/1529974415,v1:127.0.0.1:6821/1529974415] exists,up b647939a-edd3-4f8a-9297-87925944112a 2024-02-05T05:13:05.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:13:05.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:13:05.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:13:05.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-02-05T05:13:05.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-02-05T05:13:05.764 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'testpool' created 2024-02-05T05:13:05.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:13:06.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-02-05T05:13:07.289 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 1 size to 2 2024-02-05T05:13:07.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-02-05T05:13:07.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:13:07.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:13:07.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:13:07.311 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:13:07.311 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:13:07.311 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:13:07.312 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:13:07.312 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:13:07.312 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:13:07.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:13:07.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:13:07.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:13:07.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:13:07.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:13:07.423 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:13:08.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:13:08.031 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:13:08.031 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:13:08.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:13:08.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:13:08.032 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:13:08.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-02-05T05:13:08.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-02-05T05:13:08.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-02-05T05:13:08.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:13:08.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:13:08.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-02-05T05:13:08.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-02-05T05:13:08.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672973' 2024-02-05T05:13:08.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:13:08.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:13:08.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-02-05T05:13:08.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-02-05T05:13:08.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672973 2-64424509447' 2024-02-05T05:13:08.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:13:08.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-02-05T05:13:08.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:13:08.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:13:08.507 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-02-05T05:13:08.507 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:13:08.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-02-05T05:13:08.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-02-05T05:13:08.508 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836503 2024-02-05T05:13:08.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:13:08.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-02-05T05:13:08.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:13:08.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-02-05T05:13:08.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:13:08.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:13:08.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-02-05T05:13:08.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:13:08.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-02-05T05:13:08.820 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672973 2024-02-05T05:13:08.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-02-05T05:13:08.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:13:09.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-02-05T05:13:09.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:13:09.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-02-05T05:13:09.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:13:09.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:13:09.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-02-05T05:13:09.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:13:09.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-02-05T05:13:09.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-02-05T05:13:09.189 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509447 2024-02-05T05:13:09.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:13:09.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509447 2024-02-05T05:13:09.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:13:10.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:13:10.519 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:13:10.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-02-05T05:13:10.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:13:10.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:13:10.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:13:11.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:13:11.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:13:11.265 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:13:11.265 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:13:11.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:13:11.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:13:11.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:13:11.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:13:11.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T05:13:11.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:13:11.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:13:11.602 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:13:11.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T05:13:11.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-05T05:13:11.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:13:11.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-05T05:13:11.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:13:11.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:13:12.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:13:12.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:13:12.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:13:12.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:13:12.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:13:12.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:13:12.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:13:12.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:13:12.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T05:13:12.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:13:12.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:13:12.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:13:12.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T05:13:12.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T05:13:12.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:13:12.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:13:12.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:13:12.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:13:12.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:13:12.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:13:12.722 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:13:13.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:13:13.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:13:13.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:13:13.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:13:13.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:13:13.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:13:13.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:13:13.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:13:13.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:13:13.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:13:13.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:13:13.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:13:13.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:13:13.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T05:13:13.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:13:13.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:13:13.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:13:13.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T05:13:13.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T05:13:13.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:13:13.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:13:13.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:13:13.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:13:13.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:13:13.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:13:13.992 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:13:14.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:13:14.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:13:14.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T05:13:14.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:13:14.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T05:13:14.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:13:14.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:13:14.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:13:14.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:13:14.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:13:14.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:13:14.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:13:14.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:13:15.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:13:15.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:13:15.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:13:15.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:13:15.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:13:15.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:13:15.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:13:15.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-02-05T05:13:15.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-02-05T05:13:15.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-02-05T05:13:15.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T05:13:15.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:13:15.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T05:13:15.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:13:15.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T05:13:15.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:13:15.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T05:13:15.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:13:16.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T05:13:16.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:13:16.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T05:13:16.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:13:16.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T05:13:16.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:13:16.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T05:13:16.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:13:16.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T05:13:16.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:13:17.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T05:13:17.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:13:17.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-02-05T05:13:17.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:13:17.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:13:17.265 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T05:13:17.265 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:13:17.265 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:13:17.265 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:13:17.265 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:13:17.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:13:17.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:13:17.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:13:17.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-02-05T05:13:17.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:13:17.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:13:17.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:13:17.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:13:17.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-02-05T05:13:17.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:13:17.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:13:17.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:13:17.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:13:17.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:13:17.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:13:17.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:13:17.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:13:17.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:13:17.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:13:18.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:13:18.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-02-05T05:13:18.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:13:18.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:13:18.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:13:18.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:13:18.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:13:18.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-02-05T05:13:19.920 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:ff7b1f36:::obj1:head# 2024-02-05T05:13:20.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:13:20.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:13:20.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:13:20.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:13:20.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:13:20.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:13:20.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:13:20.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:13:20.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:13:20.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:13:20.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:13:20.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:13:20.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:13:20.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:13:20.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:13:20.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:13:20.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:13:20.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:13:20.459 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:13:20.459 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:13:20.459 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:13:20.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:13:20.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:13:20.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:13:20.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:13:20.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:13:20.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:13:20.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:13:20.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:13:20.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:13:20.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:13:20.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:13:20.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:13:20.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:13:20.460 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:13:20.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:13:20.461 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:13:20.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:13:20.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:13:20.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:13:20.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:13:20.496 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:20.500+0000 7fa57a42a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:20.498 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:20.504+0000 7fa57a42a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:20.499 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:20.504+0000 7fa57a42a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:20.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:13:20.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:13:20.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:13:20.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:13:20.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:13:20.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:20.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:13:20.780 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:13:20.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:20.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:13:21.047 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:21.052+0000 7fa57a42a640 -1 Falling back to public interface 2024-02-05T05:13:21.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:22.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:22.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:22.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:13:22.121 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:13:22.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:22.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:13:22.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:23.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:23.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:23.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:13:23.424 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:13:23.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:23.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:13:23.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:23.742 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:23.748+0000 7fa57a42a640 -1 osd.0 20 log_to_monitors true 2024-02-05T05:13:24.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:24.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:24.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:13:24.735 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:13:24.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:24.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:13:25.147 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/1499542060,v1:127.0.0.1:6803/1499542060] [v2:127.0.0.1:6804/1499542060,v1:127.0.0.1:6805/1499542060] exists,up a1812611-9de1-418e-b494-25a83141fa20 2024-02-05T05:13:25.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:13:25.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:13:25.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:13:25.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:13:25.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:13:25.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:13:25.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:13:25.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:13:25.149 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:13:25.149 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:13:25.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:13:25.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:13:25.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:13:25.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:13:25.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:13:25.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:13:25.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:13:25.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:13:25.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:13:25.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:13:25.578 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:13:25.578 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:13:25.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:13:25.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:13:25.579 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:13:25.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-02-05T05:13:25.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-02-05T05:13:25.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-02-05T05:13:25.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:13:25.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:13:25.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-02-05T05:13:25.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-02-05T05:13:25.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672977' 2024-02-05T05:13:25.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:13:25.894 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:13:26.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-02-05T05:13:26.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-02-05T05:13:26.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672977 2-64424509452' 2024-02-05T05:13:26.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:13:26.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-02-05T05:13:26.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:13:26.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:13:26.027 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-02-05T05:13:26.028 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:13:26.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-02-05T05:13:26.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-02-05T05:13:26.029 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 103079215106 2024-02-05T05:13:26.029 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:13:26.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 103079215106 2024-02-05T05:13:26.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:13:27.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:13:27.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:13:27.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-02-05T05:13:27.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:13:27.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-02-05T05:13:27.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:13:27.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:13:27.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-02-05T05:13:27.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:13:27.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-02-05T05:13:27.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-02-05T05:13:27.640 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672977 2024-02-05T05:13:27.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:13:27.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-02-05T05:13:27.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:13:27.970 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-02-05T05:13:27.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:13:27.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:13:27.973 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-02-05T05:13:27.974 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:13:27.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-02-05T05:13:27.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-02-05T05:13:27.975 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509452 2024-02-05T05:13:27.975 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:13:28.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-02-05T05:13:28.304 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:13:28.305 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:13:28.305 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:13:28.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:13:28.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:13:28.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:13:28.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:13:28.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:13:28.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:13:28.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:13:28.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:13:28.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:13:28.984 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:13:28.984 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:13:28.984 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:13:29.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:13:29.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:13:29.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:13:29.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-02-05T05:13:29.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:13:29.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:13:29.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:13:29.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:13:29.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-02-05T05:13:29.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:13:29.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:13:29.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:13:29.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:13:29.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-02-05T05:13:29.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:13:29.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:13:29.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:13:29.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:13:29.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:13:29.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:13:29.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:13:29.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:13:29.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:13:29.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:13:29.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:13:29.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-02-05T05:13:29.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:13:29.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:13:29.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:13:29.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:13:29.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:13:29.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-02-05T05:13:32.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:13:32.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:13:32.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:13:32.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:13:32.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:13:32.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:13:32.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:13:32.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:13:32.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:13:32.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:13:32.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:13:32.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:13:32.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:13:32.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:13:32.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:13:32.097 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:13:32.097 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:13:32.097 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:13:32.099 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:13:32.099 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:13:32.099 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:13:32.099 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:13:32.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:13:32.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:13:32.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:13:32.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:13:32.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:13:32.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:13:32.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:13:32.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:13:32.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:13:32.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:13:32.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:13:32.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:13:32.100 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:13:32.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:13:32.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:13:32.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:13:32.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:13:32.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:13:32.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:13:32.136 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:32.140+0000 7fc209815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:32.139 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:32.144+0000 7fc209815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:32.140 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:32.144+0000 7fc209815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:32.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:13:32.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:13:32.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:13:32.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:13:32.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:13:32.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:32.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:13:32.425 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:13:32.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:32.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:13:32.684 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:32.688+0000 7fc209815640 -1 Falling back to public interface 2024-02-05T05:13:32.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:33.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:33.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:33.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:13:33.787 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:13:33.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:33.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:13:34.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:35.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:35.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:35.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:13:35.104 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:13:35.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:35.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:13:35.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:35.441 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:35.448+0000 7fc209815640 -1 osd.1 25 log_to_monitors true 2024-02-05T05:13:36.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:36.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:36.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:13:36.427 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:13:36.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:36.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:13:36.788 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [10,25) [v2:127.0.0.1:6810/922538119,v1:127.0.0.1:6811/922538119] [v2:127.0.0.1:6812/922538119,v1:127.0.0.1:6813/922538119] exists,up 368b3717-439e-43b4-bd6f-31bc16e05190 2024-02-05T05:13:36.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:13:36.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:13:36.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:13:36.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:13:36.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:13:36.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:13:36.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:13:36.789 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:13:36.789 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:13:36.789 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:13:36.789 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:13:36.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:13:36.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:13:36.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:13:36.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:13:36.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:13:36.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:13:36.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:13:36.901 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:13:37.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:13:37.243 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:13:37.243 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:13:37.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:13:37.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:13:37.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:13:37.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215109 2024-02-05T05:13:37.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215109 2024-02-05T05:13:37.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109' 2024-02-05T05:13:37.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:13:37.418 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:13:37.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-02-05T05:13:37.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-02-05T05:13:37.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109 1-124554051586' 2024-02-05T05:13:37.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:13:37.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:13:37.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-02-05T05:13:37.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-02-05T05:13:37.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109 1-124554051586 2-64424509455' 2024-02-05T05:13:37.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:13:37.691 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215109 2024-02-05T05:13:37.691 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:13:37.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:13:37.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215109 2024-02-05T05:13:37.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:13:37.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215109 2024-02-05T05:13:37.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215109' 2024-02-05T05:13:37.693 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 103079215109 2024-02-05T05:13:37.694 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:13:37.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215108 -lt 103079215109 2024-02-05T05:13:37.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:13:38.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:13:38.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:13:39.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215109 2024-02-05T05:13:39.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:13:39.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:13:39.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-124554051586 2024-02-05T05:13:39.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:13:39.304 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-124554051586 2024-02-05T05:13:39.304 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:13:39.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-02-05T05:13:39.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-02-05T05:13:39.306 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 124554051586 2024-02-05T05:13:39.306 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:13:39.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-02-05T05:13:39.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:13:39.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-02-05T05:13:39.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:13:39.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:13:39.613 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-02-05T05:13:39.613 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:13:39.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-02-05T05:13:39.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-02-05T05:13:39.614 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509455 2024-02-05T05:13:39.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:13:39.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-02-05T05:13:39.894 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:13:39.894 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:13:39.894 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:13:40.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:13:40.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:13:40.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:13:40.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:13:40.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:13:40.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:13:40.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:13:40.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:13:40.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:13:40.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:13:40.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:13:40.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:13:40.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:13:40.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:13:40.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:13:40.921 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-02-05T05:13:40.921 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:13:40.921 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:13:40.921 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T05:13:40.921 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:13:40.921 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:13:40.921 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:13:40.922 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:13:41.236 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:13:41.237 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:13:41.237 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:13:41.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-02-05T05:13:41.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:13:41.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:13:41.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:13:41.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:13:41.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-02-05T05:13:41.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:13:41.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:13:41.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:13:41.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:13:41.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:13:41.551 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:13:41.551 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:13:41.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:13:41.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:13:41.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:13:41.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:13:41.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-02-05T05:13:41.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:13:41.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:13:41.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:13:41.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:13:41.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:13:41.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-02-05T05:13:43.366 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:104778fc:::obj2:head# 2024-02-05T05:13:43.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:13:43.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:13:43.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:13:43.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:13:43.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:13:43.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:13:43.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:13:43.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:13:43.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:13:43.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:13:43.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:13:43.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:13:43.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:13:43.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:13:43.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:13:43.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:13:43.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:13:43.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:13:43.903 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:13:43.903 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:13:43.904 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:13:43.904 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:13:43.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:13:43.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:13:43.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:13:43.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:13:43.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:13:43.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:13:43.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:13:43.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:13:43.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:13:43.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:13:43.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:13:43.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:13:43.905 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:13:43.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:13:43.905 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:13:43.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:13:43.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:13:43.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:13:43.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:13:43.941 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:43.944+0000 7f25409b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:43.944 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:43.948+0000 7f25409b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:43.945 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:43.952+0000 7f25409b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:44.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:13:44.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:13:44.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:13:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:13:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:13:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:13:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:13:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:13:44.480 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:44.484+0000 7f25409b0640 -1 Falling back to public interface 2024-02-05T05:13:44.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:13:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:13:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:13:45.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:46.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:46.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:46.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:13:46.889 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:13:46.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:46.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:13:47.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:47.499 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:47.504+0000 7f25409b0640 -1 osd.0 30 log_to_monitors true 2024-02-05T05:13:48.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:48.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:48.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:13:48.317 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:13:48.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:48.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:13:48.691 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 34 up_thru 26 down_at 31 last_clean_interval [24,30) [v2:127.0.0.1:6802/3130373769,v1:127.0.0.1:6803/3130373769] [v2:127.0.0.1:6804/3130373769,v1:127.0.0.1:6805/3130373769] exists,up a1812611-9de1-418e-b494-25a83141fa20 2024-02-05T05:13:48.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:13:48.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:13:48.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:13:48.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:13:48.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:13:48.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:13:48.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:13:48.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:13:48.693 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:13:48.693 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:13:48.693 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:13:48.693 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:13:48.693 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:13:48.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:13:48.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:13:48.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:13:48.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:13:48.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:13:48.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:13:49.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:13:49.140 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:13:49.140 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:13:49.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:13:49.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:13:49.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:13:49.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-02-05T05:13:49.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-02-05T05:13:49.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066' 2024-02-05T05:13:49.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:13:49.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:13:49.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-02-05T05:13:49.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-02-05T05:13:49.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-124554051589' 2024-02-05T05:13:49.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:13:49.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:13:49.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-02-05T05:13:49.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-02-05T05:13:49.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-124554051589 2-64424509458' 2024-02-05T05:13:49.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:13:49.574 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888066 2024-02-05T05:13:49.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:13:49.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:13:49.576 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888066 2024-02-05T05:13:49.576 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:13:49.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-02-05T05:13:49.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-02-05T05:13:49.577 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 146028888066 2024-02-05T05:13:49.578 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:13:49.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 146028888066 2024-02-05T05:13:49.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:13:50.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:13:50.888 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:13:51.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888066 2024-02-05T05:13:51.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:13:51.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-124554051589 2024-02-05T05:13:51.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:13:51.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:13:51.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-124554051589 2024-02-05T05:13:51.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:13:51.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-02-05T05:13:51.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 124554051589' 2024-02-05T05:13:51.209 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 124554051589 2024-02-05T05:13:51.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:13:51.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051589 2024-02-05T05:13:51.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:13:51.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-02-05T05:13:51.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:13:51.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:13:51.545 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:13:51.545 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-02-05T05:13:51.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-02-05T05:13:51.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-02-05T05:13:51.546 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509458 2024-02-05T05:13:51.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:13:51.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-02-05T05:13:51.888 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:13:51.888 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:13:51.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:13:52.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:13:52.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:13:52.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:13:52.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:13:52.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:13:52.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:13:52.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:13:52.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:13:52.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:13:52.586 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:13:52.586 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:13:52.586 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:13:52.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:13:52.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:13:52.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:13:52.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-02-05T05:13:52.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:13:52.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:13:52.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:13:52.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:13:53.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-02-05T05:13:53.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:13:53.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:13:53.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:13:53.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:13:53.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-02-05T05:13:53.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:13:53.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:13:53.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:13:53.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:13:53.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:13:53.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:13:53.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:13:53.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:13:53.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:13:53.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:13:53.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:13:53.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-02-05T05:13:53.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:13:53.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:13:53.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:13:53.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:13:53.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:13:53.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-02-05T05:13:54.971 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-02-05T05:13:55.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:13:55.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:13:55.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:13:55.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:13:55.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:13:55.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:13:55.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:13:55.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:13:55.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:13:55.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:13:55.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:13:55.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:13:55.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:13:55.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:13:55.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:13:55.519 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:13:55.519 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:13:55.519 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:13:55.519 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:13:55.520 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:13:55.520 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:13:55.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:13:55.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:13:55.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:13:55.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:13:55.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:13:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:13:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:13:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:13:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:13:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:13:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:13:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:13:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:13:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:13:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:13:55.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:13:55.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:13:55.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:13:55.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:13:55.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:13:55.557 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:55.559+0000 7f74d29b3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:55.560 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:55.563+0000 7f74d29b3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:55.561 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:55.567+0000 7f74d29b3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:55.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:13:55.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:13:55.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:13:55.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:13:55.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:13:55.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:55.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:13:55.827 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:13:55.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:55.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:13:56.096 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:56.099+0000 7f74d29b3640 -1 Falling back to public interface 2024-02-05T05:13:56.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:57.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:57.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:57.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:13:57.165 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:13:57.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:57.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:13:57.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:58.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:58.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:58.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:13:58.482 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:13:58.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:58.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:13:58.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:58.854 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:13:58.859+0000 7f74d29b3640 -1 osd.1 35 log_to_monitors true 2024-02-05T05:13:59.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:59.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:59.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:13:59.837 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:13:59.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:59.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:14:00.175 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [29,35) [v2:127.0.0.1:6810/2486096638,v1:127.0.0.1:6811/2486096638] [v2:127.0.0.1:6812/2486096638,v1:127.0.0.1:6813/2486096638] exists,up 368b3717-439e-43b4-bd6f-31bc16e05190 2024-02-05T05:14:00.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:14:00.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:14:00.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:14:00.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:14:00.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:14:00.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:14:00.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:14:00.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:14:00.177 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:14:00.177 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:14:00.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:14:00.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:14:00.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:14:00.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:14:00.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:14:00.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:14:00.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:14:00.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:14:00.285 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:14:00.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:14:00.603 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:14:00.603 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:14:00.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:14:00.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:14:00.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:14:00.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888069 2024-02-05T05:14:00.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888069 2024-02-05T05:14:00.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888069' 2024-02-05T05:14:00.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:14:00.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:14:00.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-02-05T05:14:00.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-02-05T05:14:00.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888069 1-167503724546' 2024-02-05T05:14:00.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:14:00.887 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:14:01.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-02-05T05:14:01.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-02-05T05:14:01.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888069 1-167503724546 2-64424509462' 2024-02-05T05:14:01.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:14:01.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888069 2024-02-05T05:14:01.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:14:01.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:14:01.018 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888069 2024-02-05T05:14:01.018 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:14:01.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888069 2024-02-05T05:14:01.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888069' 2024-02-05T05:14:01.019 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 146028888069 2024-02-05T05:14:01.019 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:14:01.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888069 2024-02-05T05:14:01.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:14:02.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:14:02.308 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:14:02.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888069 -lt 146028888069 2024-02-05T05:14:02.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:14:02.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724546 2024-02-05T05:14:02.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:14:02.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:14:02.642 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724546 2024-02-05T05:14:02.642 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:14:02.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-02-05T05:14:02.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2024-02-05T05:14:02.644 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 167503724546 2024-02-05T05:14:02.644 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:14:02.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-02-05T05:14:02.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:14:02.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-02-05T05:14:02.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:14:02.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:14:02.989 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-02-05T05:14:02.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:14:02.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-02-05T05:14:02.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-02-05T05:14:02.991 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509462 2024-02-05T05:14:02.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:14:03.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-02-05T05:14:03.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:14:03.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:14:03.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:14:03.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:14:03.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:14:03.694 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:14:03.694 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:14:03.694 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:14:03.694 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:14:03.694 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:14:03.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:14:03.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:14:03.974 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:14:03.974 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:14:03.974 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:14:04.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:14:04.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:14:04.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:14:04.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-02-05T05:14:04.507 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T05:14:04.507 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T05:14:04.507 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T05:14:04.520 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T05:14:04.520 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T05:14:04.520 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T05:14:04.533 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2: { 2024-02-05T05:14:04.533 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-02-05T05:14:04.533 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T05:14:04.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-02-05T05:14:04.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-02-05T05:14:04.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-02-05T05:14:04.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:14:04.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-02-05T05:14:04.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-02-05T05:14:04.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-02-05T05:14:04.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:14:04.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T05:14:04.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-02-05T05:14:04.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:14:05.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-02-05T05:14:05.198 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-02-05T05:14:05.198 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:14:05.198 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:14:05.198 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:14:05.198 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:14:05.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-02-05T05:13:05.704987+0000 2024-02-05T05:14:05.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-02-05T05:14:05.659 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T05:14:05.659 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": true, 2024-02-05T05:14:05.659 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T05:14:05.659 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-22T05:12:25.667471+0000" 2024-02-05T05:14:05.659 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T05:14:05.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-02-05T05:13:05.704987+0000 2024-02-05T05:14:05.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T05:14:05.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:13:05.704987+0000 2024-02-05T05:14:05.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:14:05.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:14:05.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:14:05.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:14:05.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:14:05.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:14:05.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:14:05.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:14:05.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:13:05.704987+0000 '>' 2024-02-05T05:13:05.704987+0000 2024-02-05T05:14:05.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:14:06.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:14:06.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:14:06.964 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:14:06.964 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:14:06.964 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:14:06.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:14:06.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:14:07.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:13:05.704987+0000 '>' 2024-02-05T05:13:05.704987+0000 2024-02-05T05:14:07.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:14:08.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:14:08.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:14:08.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:14:08.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:14:08.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:14:08.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:14:08.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:14:08.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:13:05.704987+0000 '>' 2024-02-05T05:13:05.704987+0000 2024-02-05T05:14:08.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:14:09.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:14:09.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:14:09.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:14:09.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:14:09.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:14:09.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:14:09.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:14:09.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:13:05.704987+0000 '>' 2024-02-05T05:13:05.704987+0000 2024-02-05T05:14:09.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:14:10.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:14:10.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:14:10.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:14:10.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:14:10.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:14:10.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:14:10.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:14:11.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T05:12:25.667471+0000 '>' 2024-02-05T05:13:05.704987+0000 2024-02-05T05:14:11.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:14:12.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:14:12.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:14:12.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:14:12.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:14:12.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:14:12.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:14:12.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:14:12.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T05:12:25.667471+0000 '>' 2024-02-05T05:13:05.704987+0000 2024-02-05T05:14:12.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:14:13.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:14:13.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:14:13.545 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:14:13.545 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:14:13.545 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:14:13.545 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:14:13.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:14:13.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T05:12:25.667471+0000 '>' 2024-02-05T05:13:05.704987+0000 2024-02-05T05:14:13.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:14:14.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:14:14.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:14:14.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:14:14.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:14:14.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:14:14.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:14:14.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:14:15.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:14:14.003328+0000 '>' 2024-02-05T05:13:05.704987+0000 2024-02-05T05:14:15.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:14:15.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-02-05T05:14:15.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:14:15.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:14:15.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:14:15.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:14:15.213 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:14:15.213 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:14:15.214 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:14:15.214 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:14:15.214 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:14:15.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:14:15.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:14:15.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:14:15.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:14:15.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:14:15.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:14:15.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:14:15.627 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:14:15.627 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:14:15.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:14:15.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:14:15.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:14:15.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888073 2024-02-05T05:14:15.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888073 2024-02-05T05:14:15.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888073' 2024-02-05T05:14:15.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:14:15.789 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:14:15.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724550 2024-02-05T05:14:15.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724550 2024-02-05T05:14:15.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888073 1-167503724550' 2024-02-05T05:14:15.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:14:15.929 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:14:16.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509466 2024-02-05T05:14:16.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509466 2024-02-05T05:14:16.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888073 1-167503724550 2-64424509466' 2024-02-05T05:14:16.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:14:16.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888073 2024-02-05T05:14:16.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:14:16.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:14:16.068 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888073 2024-02-05T05:14:16.068 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:14:16.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888073 2024-02-05T05:14:16.072 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 146028888073 2024-02-05T05:14:16.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888073' 2024-02-05T05:14:16.072 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:14:16.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888072 -lt 146028888073 2024-02-05T05:14:16.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:14:17.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:14:17.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:14:17.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888073 -lt 146028888073 2024-02-05T05:14:17.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:14:17.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724550 2024-02-05T05:14:17.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:14:17.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:14:17.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724550 2024-02-05T05:14:17.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:14:17.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724550 2024-02-05T05:14:17.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724550' 2024-02-05T05:14:17.690 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 167503724550 2024-02-05T05:14:17.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:14:17.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724550 -lt 167503724550 2024-02-05T05:14:17.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:14:17.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509466 2024-02-05T05:14:17.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:14:18.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:14:18.002 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509466 2024-02-05T05:14:18.002 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:14:18.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509466 2024-02-05T05:14:18.003 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509466 2024-02-05T05:14:18.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-02-05T05:14:18.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:14:18.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509466 -lt 64424509466 2024-02-05T05:14:18.317 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:14:18.318 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:14:18.318 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:14:18.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:14:18.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:14:18.711 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:14:18.711 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:14:18.711 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:14:18.711 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:14:18.711 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:14:18.712 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:14:19.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:14:19.049 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:14:19.050 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:14:19.050 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:14:19.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:14:19.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:14:19.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:14:19.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-02-05T05:14:19.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:14:19.403 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:14:19.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:14:19.718 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:14:19.718 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:14:19.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:14:19.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:14:19.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:14:19.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888075 2024-02-05T05:14:19.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888075 2024-02-05T05:14:19.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888075' 2024-02-05T05:14:19.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:14:19.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:14:20.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724552 2024-02-05T05:14:20.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724552 2024-02-05T05:14:20.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888075 1-167503724552' 2024-02-05T05:14:20.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:14:20.029 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:14:20.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-02-05T05:14:20.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-02-05T05:14:20.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888075 1-167503724552 2-64424509467' 2024-02-05T05:14:20.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:14:20.160 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888075 2024-02-05T05:14:20.160 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:14:20.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:14:20.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888075 2024-02-05T05:14:20.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:14:20.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888075 2024-02-05T05:14:20.163 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 146028888075 2024-02-05T05:14:20.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888075' 2024-02-05T05:14:20.163 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:14:20.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888074 -lt 146028888075 2024-02-05T05:14:20.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:14:21.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:14:21.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:14:21.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888075 -lt 146028888075 2024-02-05T05:14:21.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:14:21.770 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:14:21.770 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724552 2024-02-05T05:14:21.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:14:21.772 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724552 2024-02-05T05:14:21.772 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:14:21.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724552 2024-02-05T05:14:21.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724552' 2024-02-05T05:14:21.774 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 167503724552 2024-02-05T05:14:21.774 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:14:22.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724552 -lt 167503724552 2024-02-05T05:14:22.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:14:22.089 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-02-05T05:14:22.089 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:14:22.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:14:22.092 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-02-05T05:14:22.092 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:14:22.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-02-05T05:14:22.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-02-05T05:14:22.093 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509467 2024-02-05T05:14:22.094 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:14:22.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-02-05T05:14:22.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-02-05T05:14:22.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-02-05T05:14:22.798 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T05:14:22.798 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-02-05T05:14:14.003628+0000 20'10 40:98 [1,0] 1 [1,0] 1 20'10 2024-02-05T05:14:14.003328+0000 20'10 2024-02-05T05:14:14.003328+0000 0 1 periodic scrub scheduled @ 2024-02-06T05:14:14.003328+0000 10 0 2024-02-05T05:14:22.798 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T05:14:22.798 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T05:14:22.799 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:14:22.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-02-05T05:14:22.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-02-05T05:14:23.096 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:14:23.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:14:23.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:14:23.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:14:23.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:14:23.113 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:14:23.113 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:14:23.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:14:23.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:14:23.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:14:23.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:14:23.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:14:23.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:14:23.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:14:23.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:14:23.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:14:23.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:14:23.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:14:23.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:14:23.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:14:23.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:14:23.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:14:23.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:14:23.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:14:23.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:14:23.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:14:23.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:14:23.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:14:23.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:14:23.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:14:23.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:14:23.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:14:23.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:14:23.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:14:23.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:14:23.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:14:23.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:14:23.250 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:14:23.250 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:14:23.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:14:23.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:14:23.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:14:23.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:14:23.252 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:14:23.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:14:23.253 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:14:23.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:14:23.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:14:23.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:14:23.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:14:23.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:14:23.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:14:23.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:14:23.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:14:23.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:14:23.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:14:23.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:14:23.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:14:23.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:14:23.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:14:23.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:14:23.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:14:23.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:14:23.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:14:23.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:14:23.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:14:23.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T05:14:23.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:14:23.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:14:23.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:14:23.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:14:23.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-02-05T05:14:23.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-02-05T05:14:23.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-02-05T05:14:23.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-02-05T05:14:23.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:14:23.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:14:23.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:14:23.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:14:23.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:14:23.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:14:23.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:14:23.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:14:23.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:14:23.676 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:14:23.676 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:14:23.676 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:14:23.676 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:14:23.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:14:23.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:14:23.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:14:23.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:14:23.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:14:23.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:14:23.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:14:23.899 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:14:23.899 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:14:23.899 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:14:23.900 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:14:23.900 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:14:23.900 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:14:23.900 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:14:23.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:14:23.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T05:14:24.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:14:24.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:14:24.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:14:24.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:14:24.018 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:14:24.018 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:14:24.019 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:14:24.019 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:14:24.019 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:14:24.019 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:14:24.019 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:14:24.019 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:14:24.019 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:14:24.019 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T05:14:24.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-02-05T05:14:24.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:14:24.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:14:24.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:14:24.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:14:24.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:14:24.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:14:24.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:14:24.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:14:24.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:14:24.996 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:14:24.996 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:14:24.996 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:14:24.996 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:14:24.997 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:14:24.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:14:25.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-02-05T05:14:25.033 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-02-05T05:14:25.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-02-05T05:14:25.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T05:14:25.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:14:25.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:14:25.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:14:25.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:14:25.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:14:25.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:14:25.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:14:25.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:14:25.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:14:25.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:14:25.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:14:25.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:14:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:14:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:14:25.038 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:14:25.038 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:14:25.038 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:14:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:14:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:14:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:14:25.038 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:14:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:14:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:14:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:14:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:14:25.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:14:25.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:14:25.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:14:25.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:14:25.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:14:25.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:14:25.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:14:25.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-02-05T05:14:25.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:14:25.040 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:14:25.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c2c71d8-544f-47c4-9907-ee26a494f1fd 2024-02-05T05:14:25.040 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 5c2c71d8-544f-47c4-9907-ee26a494f1fd 2024-02-05T05:14:25.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5c2c71d8-544f-47c4-9907-ee26a494f1fd' 2024-02-05T05:14:25.040 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:14:25.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxbsBlEujJAxAAQ6gm6L7RR2VojjDsl7cvbQ== 2024-02-05T05:14:25.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxbsBlEujJAxAAQ6gm6L7RR2VojjDsl7cvbQ=="}' 2024-02-05T05:14:25.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c2c71d8-544f-47c4-9907-ee26a494f1fd -i td/osd-scrub-repair/0/new.json 2024-02-05T05:14:25.324 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:14:25.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:14:25.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCxbsBlEujJAxAAQ6gm6L7RR2VojjDsl7cvbQ== --osd-uuid 5c2c71d8-544f-47c4-9907-ee26a494f1fd 2024-02-05T05:14:25.379 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:14:25.382+0000 7fdaddfbe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:25.391 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:14:25.394+0000 7fdaddfbe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:25.393 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:14:25.394+0000 7fdaddfbe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:25.393 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:14:25.394+0000 7fdaddfbe640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:14:25.393 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:14:25.394+0000 7fdaddfbe640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:14:25.394 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:14:25.394+0000 7fdaddfbe640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:14:25.492 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:14:25.494+0000 7fdaddfbe640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:14:32.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:14:32.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:14:32.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:14:32.097 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:14:32.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:14:32.470 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:14:32.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:14:32.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T05:14:32.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:14:32.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:14:32.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:14:32.505 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:14:32.510+0000 7fcdac9af640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:32.541 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:14:32.546+0000 7fcdac9af640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:32.543 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:14:32.550+0000 7fcdac9af640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:32.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:14:32.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:14:32.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:14:32.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:14:32.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:14:32.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:32.798 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:14:32.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:14:32.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:32.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:33.100 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:14:33.106+0000 7fcdac9af640 -1 Falling back to public interface 2024-02-05T05:14:33.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:34.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:34.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:34.134 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:14:34.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:14:34.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:34.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:34.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:35.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:35.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:35.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:14:35.447 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:14:35.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:35.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:35.692 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:14:35.698+0000 7fcdac9af640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:14:35.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:36.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:36.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:36.755 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:14:36.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:14:36.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:36.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:37.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:38.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:38.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:38.067 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:14:38.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:14:38.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:38.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:38.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:39.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:39.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:39.401 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:14:39.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:14:39.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:39.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:39.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:40.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:40.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:40.768 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:14:40.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:14:40.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:40.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:41.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:42.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:42.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:42.071 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:14:42.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:14:42.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:42.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:42.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:43.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:43.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:43.384 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:14:43.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:14:43.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:43.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:43.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:44.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:44.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:44.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:14:44.771 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:14:44.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:44.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:45.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:46.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:46.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:46.073 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:14:46.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:14:46.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:46.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:46.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:47.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:47.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:47.380 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:14:47.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:14:47.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:47.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:47.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:48.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:48.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:48.774 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:14:48.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:14:48.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:48.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:49.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:50.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:50.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:50.079 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:14:50.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:14:50.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:50.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:50.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:51.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:51.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:51.488 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:14:51.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:14:51.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:51.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:52.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:52.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:52.984 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:14:52.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:14:52.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:52.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:53.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:54.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:54.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:54.582 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:14:54.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:14:54.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:54.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:55.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:56.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:56.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:56.076 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:14:56.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:14:56.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:56.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:56.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:57.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:57.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:57.688 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:14:57.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:14:57.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:57.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:58.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:59.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:59.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:59.223 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:14:59.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:14:59.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:59.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:14:59.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:00.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:00.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:00.826 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:15:00.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:15:00.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:00.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:15:01.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:02.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:02.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:02.145 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T05:15:02.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T05:15:02.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:02.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:15:02.458 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/508040956,v1:127.0.0.1:6803/508040956] [v2:127.0.0.1:6804/508040956,v1:127.0.0.1:6805/508040956] exists,up 5c2c71d8-544f-47c4-9907-ee26a494f1fd 2024-02-05T05:15:02.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:15:02.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:15:02.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:15:02.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-02-05T05:15:02.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T05:15:02.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:15:02.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:15:02.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:15:02.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:15:02.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:15:02.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:15:02.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:15:02.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:15:02.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:15:02.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:15:02.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:15:02.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:15:02.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:15:02.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:15:02.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:15:02.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:15:02.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:15:02.463 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:15:02.463 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:15:02.463 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:15:02.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:15:02.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:15:02.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:15:02.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:15:02.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:15:02.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:15:02.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:15:02.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:15:02.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:15:02.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:15:02.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:15:02.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:15:02.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-02-05T05:15:02.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:15:02.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:15:02.465 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 7eccd6e1-33e4-4d29-8436-e8b8751badc4 2024-02-05T05:15:02.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7eccd6e1-33e4-4d29-8436-e8b8751badc4 2024-02-05T05:15:02.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7eccd6e1-33e4-4d29-8436-e8b8751badc4' 2024-02-05T05:15:02.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:15:02.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWbsBlwYQkHRAA/jT5xlSia7xrLuod0JNyfg== 2024-02-05T05:15:02.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWbsBlwYQkHRAA/jT5xlSia7xrLuod0JNyfg=="}' 2024-02-05T05:15:02.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7eccd6e1-33e4-4d29-8436-e8b8751badc4 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:15:02.839 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:15:02.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:15:02.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDWbsBlwYQkHRAA/jT5xlSia7xrLuod0JNyfg== --osd-uuid 7eccd6e1-33e4-4d29-8436-e8b8751badc4 2024-02-05T05:15:02.895 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:02.901+0000 7fdc0aff7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:02.924 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:02.929+0000 7fdc0aff7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:02.926 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:02.929+0000 7fdc0aff7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:02.926 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:02.933+0000 7fdc0aff7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:15:02.926 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:02.933+0000 7fdc0aff7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:15:02.926 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:02.933+0000 7fdc0aff7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:15:02.998 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:03.001+0000 7fdc0aff7640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:15:10.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:15:10.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:15:10.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:15:10.138 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:15:10.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:15:10.518 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:15:10.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:15:10.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:15:10.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T05:15:10.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:15:10.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:15:10.552 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:10.556+0000 7faeb5254640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:10.581 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:10.584+0000 7faeb5254640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:10.582 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:10.584+0000 7faeb5254640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:10.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:15:10.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:15:10.851 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:15:10.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:15:10.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:15:10.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:15:10.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:10.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:15:10.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:10.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:11.129 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:11.132+0000 7faeb5254640 -1 Falling back to public interface 2024-02-05T05:15:11.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:12.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:12.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:12.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:15:12.173 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:15:12.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:12.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:12.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:13.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:13.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:13.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:15:13.490 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:15:13.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:13.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:13.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:13.815 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:13.820+0000 7faeb5254640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:15:14.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:14.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:14.808 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:15:14.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:15:14.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:14.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:15.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:16.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:16.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:16.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:15:16.127 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:15:16.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:16.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:16.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:17.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:17.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:17.453 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:15:17.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:15:17.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:17.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:18.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:19.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:19.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:19.009 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:15:19.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:15:19.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:19.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:19.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:20.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:20.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:20.322 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:15:20.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:15:20.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:20.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:20.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:21.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:21.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:21.818 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:15:21.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:15:21.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:21.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:22.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:23.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:23.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:23.194 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:15:23.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:15:23.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:23.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:23.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:24.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:24.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:24.542 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:15:24.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:15:24.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:24.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:25.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:26.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:26.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:26.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:15:26.018 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:15:26.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:26.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:26.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:27.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:27.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:27.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:15:27.431 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:15:27.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:27.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:27.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:28.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:28.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:28.759 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:15:28.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:15:28.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:28.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:29.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:30.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:30.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:30.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:15:30.353 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:15:30.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:30.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:30.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:31.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:31.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:31.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:15:31.946 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:15:31.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:31.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:32.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:33.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:33.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:33.368 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:15:33.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:15:33.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:33.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:33.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:34.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:34.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:34.811 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:15:34.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:15:34.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:34.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:35.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:36.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:36.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:36.423 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:15:36.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:15:36.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:36.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:36.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:37.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:37.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:37.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:15:37.806 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:15:37.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:37.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:38.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:39.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:39.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:39.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:15:39.196 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:15:39.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:39.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:39.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:40.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:40.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:40.580 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T05:15:40.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T05:15:40.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:40.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:15:40.892 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3366485770,v1:127.0.0.1:6811/3366485770] [v2:127.0.0.1:6812/3366485770,v1:127.0.0.1:6813/3366485770] exists,up 7eccd6e1-33e4-4d29-8436-e8b8751badc4 2024-02-05T05:15:40.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:15:40.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:15:40.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:15:40.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-02-05T05:15:40.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T05:15:40.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:15:40.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:15:40.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:15:40.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:15:40.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:15:40.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:15:40.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:15:40.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:15:40.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:15:40.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:15:40.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:15:40.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:15:40.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:15:40.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:15:40.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:15:40.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:15:40.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:15:40.898 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:15:40.898 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:15:40.898 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:15:40.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:15:40.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:15:40.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:15:40.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:15:40.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:15:40.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:15:40.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:15:40.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:15:40.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:15:40.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:15:40.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:15:40.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:15:40.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-02-05T05:15:40.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:15:40.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:15:40.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=821719af-53f2-4c03-8f3d-a394f8dc155d 2024-02-05T05:15:40.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 821719af-53f2-4c03-8f3d-a394f8dc155d' 2024-02-05T05:15:40.900 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 821719af-53f2-4c03-8f3d-a394f8dc155d 2024-02-05T05:15:40.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:15:40.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8bsBlaI/fNhAAmlT4Tp0ma2X4m1aQ/ZW1Gg== 2024-02-05T05:15:40.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8bsBlaI/fNhAAmlT4Tp0ma2X4m1aQ/ZW1Gg=="}' 2024-02-05T05:15:40.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 821719af-53f2-4c03-8f3d-a394f8dc155d -i td/osd-scrub-repair/2/new.json 2024-02-05T05:15:41.298 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:15:41.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:15:41.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQD8bsBlaI/fNhAAmlT4Tp0ma2X4m1aQ/ZW1Gg== --osd-uuid 821719af-53f2-4c03-8f3d-a394f8dc155d 2024-02-05T05:15:41.356 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:41.359+0000 7fac0c63a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:41.408 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:41.411+0000 7fac0c63a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:41.409 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:41.411+0000 7fac0c63a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:41.410 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:41.411+0000 7fac0c63a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:15:41.410 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:41.411+0000 7fac0c63a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:15:41.410 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:41.411+0000 7fac0c63a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:15:41.506 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:41.507+0000 7fac0c63a640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:15:48.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:15:48.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:15:48.230 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T05:15:48.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:15:48.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:15:48.604 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T05:15:48.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:15:48.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T05:15:48.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:15:48.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:15:48.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:15:48.640 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:48.643+0000 7fb3fcc25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:48.673 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:48.675+0000 7fb3fcc25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:48.674 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:48.675+0000 7fb3fcc25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:48.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:15:48.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:15:48.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:15:48.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:15:48.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:15:48.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:48.960 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:15:48.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:15:48.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:48.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:15:49.230 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:49.231+0000 7fb3fcc25640 -1 Falling back to public interface 2024-02-05T05:15:49.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:50.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:50.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:50.295 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:15:50.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:15:50.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:50.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:15:50.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:51.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:51.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:51.607 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:15:51.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:15:51.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:51.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:15:51.846 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:15:51.847+0000 7fb3fcc25640 -1 osd.2 0 log_to_monitors true 2024-02-05T05:15:51.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:52.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:52.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:52.916 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:15:52.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:15:52.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:52.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:15:53.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:54.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:54.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:54.242 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:15:54.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:15:54.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:54.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:15:54.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:55.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:55.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:55.608 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:15:55.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:15:55.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:55.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:15:55.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:56.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:56.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:56.928 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:15:56.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:15:56.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:56.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:15:57.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:58.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:58.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:58.251 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:15:58.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:15:58.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:58.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:15:58.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:59.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:59.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:59.601 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:15:59.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:15:59.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:59.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:15:59.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:00.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:00.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:00.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:16:00.960 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:16:00.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:00.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:16:01.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:02.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:02.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:02.283 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:16:02.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:16:02.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:02.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:16:02.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:03.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:03.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:03.593 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:16:03.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:16:03.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:03.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:16:03.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:04.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:04.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:04.945 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:16:04.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:16:04.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:04.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:16:05.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:06.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:06.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:06.263 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:16:06.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:16:06.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:06.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:16:06.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:07.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:07.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:07.745 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:16:07.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:16:07.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:07.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:16:08.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:09.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:09.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:09.248 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:16:09.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:16:09.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:09.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:16:09.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:10.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:10.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:10.748 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:16:10.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:16:10.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:10.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:16:11.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:12.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:12.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:12.489 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:16:12.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:16:12.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:12.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:16:12.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:13.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:13.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:13.999 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:16:14.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:16:14.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:14.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:16:14.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:15.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:15.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:15.784 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:16:15.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:16:15.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:15.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:16:16.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:17.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:17.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:17.453 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:16:17.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:16:17.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:17.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:16:17.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:18.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:18.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:18.987 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T05:16:18.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T05:16:18.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:18.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:16:19.302 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3833440937,v1:127.0.0.1:6819/3833440937] [v2:127.0.0.1:6820/3833440937,v1:127.0.0.1:6821/3833440937] exists,up 821719af-53f2-4c03-8f3d-a394f8dc155d 2024-02-05T05:16:19.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:16:19.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:16:19.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:16:19.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-02-05T05:16:19.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-02-05T05:16:19.791 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'testpool' created 2024-02-05T05:16:19.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:16:20.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-02-05T05:16:21.356 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 1 size to 2 2024-02-05T05:16:21.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-02-05T05:16:21.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:16:21.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:16:21.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:16:21.382 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:16:21.383 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:16:21.383 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:16:21.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:16:21.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:16:21.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:16:21.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:16:21.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:16:21.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:16:21.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:16:21.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:16:21.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:16:21.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:16:21.869 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:16:21.870 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:16:21.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:16:21.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:21.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:16:22.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-02-05T05:16:22.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-02-05T05:16:22.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-02-05T05:16:22.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:22.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:16:22.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-02-05T05:16:22.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-02-05T05:16:22.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672975' 2024-02-05T05:16:22.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:22.390 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:16:22.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509443 2024-02-05T05:16:22.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509443 2024-02-05T05:16:22.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672975 2-64424509443' 2024-02-05T05:16:22.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:22.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-02-05T05:16:22.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:22.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:16:22.541 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-02-05T05:16:22.541 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:22.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-02-05T05:16:22.542 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836501 2024-02-05T05:16:22.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-02-05T05:16:22.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:16:22.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-02-05T05:16:22.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:16:23.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:16:23.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:16:24.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836501 2024-02-05T05:16:24.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:24.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-02-05T05:16:24.166 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:24.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:16:24.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-02-05T05:16:24.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:24.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-02-05T05:16:24.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-02-05T05:16:24.170 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672975 2024-02-05T05:16:24.170 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:16:24.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-02-05T05:16:24.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:24.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509443 2024-02-05T05:16:24.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:24.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:16:24.519 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509443 2024-02-05T05:16:24.519 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:24.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509443 2024-02-05T05:16:24.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2024-02-05T05:16:24.521 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509443 2024-02-05T05:16:24.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:16:24.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509443 -lt 64424509443 2024-02-05T05:16:24.850 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:16:24.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:16:24.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:16:25.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:16:25.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:16:25.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:16:25.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:16:25.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:16:25.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:16:25.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:16:25.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:16:25.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:16:25.627 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:16:25.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:16:25.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:16:26.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:16:26.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:16:26.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:16:26.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-02-05T05:16:26.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-02-05T05:16:26.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:16:26.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-02-05T05:16:26.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:16:26.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:16:26.185 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T05:16:26.185 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:16:26.185 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:16:26.185 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:16:26.185 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:16:26.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:16:26.565 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:16:26.565 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:16:26.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:16:26.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:16:26.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:16:26.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:16:26.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:16:26.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:16:26.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:16:26.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:16:26.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:16:26.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:16:26.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:16:26.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:16:26.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:16:26.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:16:26.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:16:26.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:16:27.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:16:27.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:16:27.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:16:27.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:16:27.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:16:27.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:16:27.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:16:27.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-02-05T05:16:28.908 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-02-05T05:16:29.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T05:16:29.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:16:29.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:16:29.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:16:29.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:16:29.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:16:29.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:16:29.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:16:29.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:16:29.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:16:29.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:16:29.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:16:29.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:16:29.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:16:29.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:16:29.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:16:29.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:16:29.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:16:29.450 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:16:29.450 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:16:29.450 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:16:29.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:16:29.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:16:29.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:16:29.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:16:29.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:16:29.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:16:29.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:16:29.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:16:29.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:16:29.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:16:29.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-02-05T05:16:29.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:16:29.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:16:29.452 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:16:29.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T05:16:29.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:16:29.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:16:29.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:16:29.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:16:29.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:16:29.493 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:16:29.497+0000 7f0d45276640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:16:29.495 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:16:29.497+0000 7f0d45276640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:16:29.497 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:16:29.501+0000 7f0d45276640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:16:29.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:16:29.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:16:29.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:16:29.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:16:29.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:16:29.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:29.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:16:29.770 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:16:29.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:29.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:16:30.046 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:16:30.049+0000 7f0d45276640 -1 Falling back to public interface 2024-02-05T05:16:30.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:31.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:31.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:31.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:16:31.117 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:16:31.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:31.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:16:31.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:32.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:16:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:16:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:16:32.456 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:16:32.461+0000 7f0d45276640 -1 osd.0 21 log_to_monitors true 2024-02-05T05:16:32.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:33.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:33.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:33.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:16:33.771 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:16:33.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:33.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:16:34.158 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1684700740,v1:127.0.0.1:6803/1684700740] [v2:127.0.0.1:6804/1684700740,v1:127.0.0.1:6805/1684700740] exists,up 5c2c71d8-544f-47c4-9907-ee26a494f1fd 2024-02-05T05:16:34.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:16:34.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:16:34.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:16:34.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:16:34.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:16:34.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:16:34.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:16:34.159 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:16:34.160 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:16:34.160 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:16:34.160 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:16:34.160 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:16:34.160 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:16:34.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:16:34.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:16:34.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:16:34.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:16:34.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:16:34.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:16:34.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:16:34.590 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:16:34.590 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:16:34.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:16:34.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:34.591 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:16:34.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-02-05T05:16:34.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-02-05T05:16:34.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-02-05T05:16:34.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:34.770 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:16:34.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-02-05T05:16:34.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-02-05T05:16:34.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978' 2024-02-05T05:16:34.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:34.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:16:35.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-02-05T05:16:35.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-02-05T05:16:35.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978 2-64424509447' 2024-02-05T05:16:35.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:35.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-02-05T05:16:35.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:35.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:16:35.054 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-02-05T05:16:35.054 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:35.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-02-05T05:16:35.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-02-05T05:16:35.055 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 107374182402 2024-02-05T05:16:35.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:16:35.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-02-05T05:16:35.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:16:36.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:16:36.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:16:36.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-02-05T05:16:36.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:36.669 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-02-05T05:16:36.669 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:36.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:16:36.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-02-05T05:16:36.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:36.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-02-05T05:16:36.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-02-05T05:16:36.673 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672978 2024-02-05T05:16:36.673 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:16:37.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672978 2024-02-05T05:16:37.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:37.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:37.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-02-05T05:16:37.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:16:37.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-02-05T05:16:37.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:37.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-02-05T05:16:37.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-02-05T05:16:37.008 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509447 2024-02-05T05:16:37.008 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:16:37.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-02-05T05:16:37.344 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:16:37.344 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:16:37.344 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:16:37.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:16:37.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:16:37.762 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:16:37.762 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:16:37.762 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:16:37.763 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:16:37.763 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:16:37.763 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:16:38.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:16:38.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:16:38.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:16:38.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:16:38.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:16:38.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:16:38.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:16:38.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-02-05T05:16:38.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-02-05T05:16:38.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:16:38.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-02-05T05:16:38.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:16:38.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-02-05T05:16:38.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-02-05T05:16:38.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:16:38.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:16:38.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:16:38.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:16:39.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-02-05T05:16:39.093 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-02-05T05:16:39.093 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:16:39.093 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:16:39.094 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:16:39.094 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:16:39.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-02-05T05:16:19.671306+0000 2024-02-05T05:16:39.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-02-05T05:16:39.516 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T05:16:39.516 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T05:16:39.516 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T05:16:39.516 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-29T05:14:59.522142+0000" 2024-02-05T05:16:39.516 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T05:16:39.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-02-05T05:16:19.671306+0000 2024-02-05T05:16:39.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T05:16:39.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:16:19.671306+0000 2024-02-05T05:16:39.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:16:39.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:16:39.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:16:39.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:16:39.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:16:39.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:16:39.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:16:39.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:16:39.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:16:19.671306+0000 '>' 2024-02-05T05:16:19.671306+0000 2024-02-05T05:16:39.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:16:40.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:16:40.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:16:40.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:16:40.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:16:40.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:16:40.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:16:40.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:16:41.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:16:19.671306+0000 '>' 2024-02-05T05:16:19.671306+0000 2024-02-05T05:16:41.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:16:42.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:16:42.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:16:42.116 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:16:42.116 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:16:42.116 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:16:42.117 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:16:42.117 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:16:42.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-29T05:14:59.522142+0000 '>' 2024-02-05T05:16:19.671306+0000 2024-02-05T05:16:42.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:16:43.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:16:43.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:16:43.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:16:43.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:16:43.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:16:43.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:16:43.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:16:43.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-29T05:14:59.522142+0000 '>' 2024-02-05T05:16:19.671306+0000 2024-02-05T05:16:43.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:16:44.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:16:44.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:16:44.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:16:44.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:16:44.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:16:44.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:16:44.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:16:45.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-29T05:14:59.522142+0000 '>' 2024-02-05T05:16:19.671306+0000 2024-02-05T05:16:45.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:16:46.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:16:46.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:16:46.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:16:46.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:16:46.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:16:46.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:16:46.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:16:46.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:16:41.519497+0000 '>' 2024-02-05T05:16:19.671306+0000 2024-02-05T05:16:46.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:16:46.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-02-05T05:16:46.689 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T05:16:46.689 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-02-05T05:16:41.730049+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-02-05T05:16:41.519497+0000 21'1 2024-02-05T05:16:41.519497+0000 0 1 periodic scrub scheduled @ 2024-02-06T05:16:41.519497+0000 1 0 2024-02-05T05:16:46.689 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T05:16:46.689 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T05:16:46.690 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:16:46.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-02-05T05:16:48.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-02-05T05:16:48.990 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T05:16:48.991 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-02-05T05:16:41.730049+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-02-05T05:16:41.519497+0000 21'1 2024-02-05T05:16:41.519497+0000 0 1 periodic scrub scheduled @ 2024-02-06T05:16:41.519497+0000 1 0 2024-02-05T05:16:48.991 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T05:16:48.991 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T05:16:48.991 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:16:49.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-02-05T05:16:51.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-02-05T05:16:51.305 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T05:16:51.305 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-02-05T05:16:41.730049+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-02-05T05:16:41.519497+0000 21'1 2024-02-05T05:16:41.519497+0000 0 1 periodic scrub scheduled @ 2024-02-06T05:16:41.519497+0000 1 0 2024-02-05T05:16:51.305 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T05:16:51.305 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T05:16:51.306 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:16:51.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-02-05T05:16:56.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-02-05T05:16:56.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:16:56.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:16:56.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:16:56.325 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:16:56.326 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:16:56.326 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:16:56.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:16:56.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:16:56.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:16:56.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:16:56.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:16:56.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:16:56.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:16:56.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:16:56.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:16:56.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:16:56.768 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:16:56.768 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:16:56.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:16:56.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:56.769 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:16:56.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182407 2024-02-05T05:16:56.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182407 2024-02-05T05:16:56.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407' 2024-02-05T05:16:56.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:56.929 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:16:57.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-02-05T05:16:57.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-02-05T05:16:57.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672984' 2024-02-05T05:16:57.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:57.075 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:16:57.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-02-05T05:16:57.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-02-05T05:16:57.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672984 2-64424509452' 2024-02-05T05:16:57.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:57.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182407 2024-02-05T05:16:57.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:57.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:16:57.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182407 2024-02-05T05:16:57.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:57.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182407 2024-02-05T05:16:57.220 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 107374182407 2024-02-05T05:16:57.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-02-05T05:16:57.221 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:16:57.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182407 2024-02-05T05:16:57.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:16:58.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:16:58.526 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:16:58.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182407 -lt 107374182407 2024-02-05T05:16:58.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:58.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:58.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-02-05T05:16:58.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:16:58.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-02-05T05:16:58.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:58.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-02-05T05:16:58.834 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672984 2024-02-05T05:16:58.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-02-05T05:16:58.834 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:16:59.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-02-05T05:16:59.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:59.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-02-05T05:16:59.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:59.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:16:59.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-02-05T05:16:59.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:59.191 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509452 2024-02-05T05:16:59.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-02-05T05:16:59.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-02-05T05:16:59.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:16:59.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-02-05T05:16:59.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:16:59.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:16:59.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:16:59.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:16:59.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:16:59.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:16:59.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:16:59.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:16:59.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:16:59.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:16:59.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:17:00.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:17:00.269 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:17:00.269 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:17:00.269 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:17:00.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:17:00.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:17:00.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:17:00.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-02-05T05:17:00.955 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T05:17:00.955 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-02-05T05:16:41.730049+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-02-05T05:16:41.519497+0000 21'1 2024-02-05T05:16:41.519497+0000 0 1 periodic scrub scheduled @ 2024-02-06T05:16:41.519497+0000 1 0 2024-02-05T05:17:00.955 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T05:17:00.955 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T05:17:00.955 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:17:00.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-02-05T05:17:00.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:17:00.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:17:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T05:17:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:17:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:17:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:17:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:17:01.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:17:01.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:17:01.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:17:01.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T05:17:01.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:17:01.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:17:01.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:17:01.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:17:01.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T05:17:01.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:17:01.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:17:01.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:17:01.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:17:01.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:17:01.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:17:01.590 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:17:01.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:17:01.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:17:01.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:17:01.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:17:01.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T05:17:01.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:17:01.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:17:01.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:17:01.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:17:01.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:17:01.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-02-05T05:17:03.070 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:17:03.070 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:17:03.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T05:17:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:17:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:17:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:17:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:17:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:17:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:17:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:17:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:17:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:17:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:17:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:17:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:17:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:17:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:17:03.372 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:17:03.372 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:17:03.372 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:17:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:17:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:17:03.374 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:17:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:17:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:17:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:17:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:17:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:17:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:17:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:17:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:17:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:17:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-02-05T05:17:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:17:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:17:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:17:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T05:17:03.375 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:17:03.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:17:03.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:17:03.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:17:03.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:17:03.410 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:03.412+0000 7efec4923640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:03.413 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:03.416+0000 7efec4923640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:03.415 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:03.416+0000 7efec4923640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:03.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:17:03.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:17:03.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:17:03.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:17:03.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:17:03.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:03.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:17:03.711 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:17:03.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:03.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:17:03.970 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:03.972+0000 7efec4923640 -1 Falling back to public interface 2024-02-05T05:17:04.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:05.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:05.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:05.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:17:05.009 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:17:05.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:05.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:17:05.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:06.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:06.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:06.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:17:06.326 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:17:06.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:06.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:17:06.657 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:06.660+0000 7efec4923640 -1 osd.0 26 log_to_monitors true 2024-02-05T05:17:06.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:07.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:07.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:07.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:17:07.696 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:17:07.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:07.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:17:08.109 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/1911292247,v1:127.0.0.1:6803/1911292247] [v2:127.0.0.1:6804/1911292247,v1:127.0.0.1:6805/1911292247] exists,up 5c2c71d8-544f-47c4-9907-ee26a494f1fd 2024-02-05T05:17:08.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:17:08.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:17:08.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:17:08.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:17:08.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:17:08.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:17:08.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:17:08.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:17:08.110 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:17:08.110 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:17:08.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:17:08.111 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:17:08.111 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:17:08.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:17:08.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:17:08.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:17:08.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:17:08.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:17:08.214 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:17:08.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:17:08.539 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:17:08.539 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:17:08.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:17:08.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:08.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:17:08.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-02-05T05:17:08.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-02-05T05:17:08.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-02-05T05:17:08.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:08.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:17:08.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672987 2024-02-05T05:17:08.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672987 2024-02-05T05:17:08.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672987' 2024-02-05T05:17:08.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:08.863 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:17:09.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-02-05T05:17:09.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-02-05T05:17:09.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672987 2-64424509455' 2024-02-05T05:17:09.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:09.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-02-05T05:17:09.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:09.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:17:09.008 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-02-05T05:17:09.008 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:09.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-02-05T05:17:09.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-02-05T05:17:09.010 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 128849018882 2024-02-05T05:17:09.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:17:09.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-02-05T05:17:09.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:17:10.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:17:10.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:17:10.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-02-05T05:17:10.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:10.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672987 2024-02-05T05:17:10.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:10.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:17:10.644 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672987 2024-02-05T05:17:10.644 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:10.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672987 2024-02-05T05:17:10.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-02-05T05:17:10.645 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672987 2024-02-05T05:17:10.646 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:17:10.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672987 2024-02-05T05:17:10.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:10.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-02-05T05:17:10.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:10.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:17:10.962 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-02-05T05:17:10.963 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:10.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-02-05T05:17:10.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-02-05T05:17:10.964 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509455 2024-02-05T05:17:10.964 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:17:11.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-02-05T05:17:11.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:17:11.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:17:11.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:17:11.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:17:11.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:17:11.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:17:11.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:17:11.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:17:11.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:17:11.693 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:17:11.693 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:17:12.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:17:12.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:17:12.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:17:12.012 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:17:12.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:17:12.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:17:12.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:17:12.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:17:12.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:17:12.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-02-05T05:17:12.435 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:16:40.469+0000 7fae991de640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] planned DEEP_SCRUB_ON_ERROR] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-02-05T05:17:12.435 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:16:40.469+0000 7fae991de640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-02-05T05:17:12.435 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:16:40.469+0000 7fae991de640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-02-05T05:17:12.435 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:16:41.517+0000 7fae991de640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T05:17:12.435 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:16:41.517+0000 7fae991de640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-02-05T05:17:12.435 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:16:41.517+0000 7fae991de640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-02-05T05:17:12.435 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:16:41.517+0000 7fae991de640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-02-05T05:17:12.435 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:16:41.517+0000 7fae991de640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T05:17:12.436 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-02-05T05:17:12.436 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-02-05T05:17:12.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-02-05T05:17:12.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-02-05T05:17:12.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:17:12.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:17:12.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:17:12.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:17:12.577 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:17:12.577 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:17:12.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:17:12.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:17:12.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:17:12.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:17:12.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:17:12.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:17:12.687 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:17:12.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:17:12.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:17:12.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:17:12.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:17:12.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:17:12.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:17:12.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:17:12.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:17:12.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:17:12.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:17:12.707 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:17:12.707 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:12.707 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:17:12.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:17:12.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:17:12.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:17:12.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:17:12.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:17:12.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:17:12.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:17:12.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:17:12.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:17:12.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:17:12.710 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:17:12.710 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:17:12.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:17:12.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:17:12.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:17:12.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:17:12.711 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:17:12.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:17:12.712 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:17:12.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:17:12.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:17:12.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:17:12.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:17:12.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:17:12.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:17:12.715 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:17:12.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:17:12.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:17:12.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:17:12.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:17:12.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:12.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:17:12.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:17:12.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:17:12.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:17:12.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:17:12.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:17:12.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:12.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:17:12.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T05:17:12.722 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:17:12.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:17:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:17:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:17:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-02-05T05:17:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-02-05T05:17:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-02-05T05:17:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-02-05T05:17:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:17:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:17:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:17:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:17:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:17:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:17:13.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:17:13.116 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:17:13.116 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:17:13.116 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:17:13.116 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:13.117 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:17:13.117 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:17:13.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:17:13.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:17:13.346 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:17:13.346 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:17:13.346 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:17:13.346 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:17:13.347 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:17:13.347 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:17:13.347 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:17:13.347 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:17:13.347 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:17:13.348 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:13.348 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:17:13.348 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:17:13.348 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:17:13.348 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T05:17:13.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:17:13.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:17:13.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:17:13.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:17:13.462 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:17:13.462 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:17:13.462 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:17:13.462 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:17:13.462 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:17:13.462 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:13.462 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:17:13.462 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:17:13.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:17:13.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T05:17:13.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-02-05T05:17:13.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:17:13.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:17:13.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:17:13.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:17:13.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:17:13.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:17:14.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:17:14.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:17:14.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:17:14.482 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:17:14.482 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:14.483 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:17:14.483 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:17:14.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:17:14.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:17:14.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-02-05T05:17:14.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-02-05T05:17:14.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-02-05T05:17:14.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T05:17:14.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:17:14.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:17:14.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:17:14.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:17:14.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:17:14.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:17:14.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:17:14.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:17:14.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:17:14.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:17:14.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:17:14.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:17:14.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:17:14.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:17:14.527 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:17:14.527 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:17:14.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:17:14.528 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:17:14.528 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:14.528 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:17:14.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:17:14.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:17:14.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:17:14.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:17:14.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:17:14.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:17:14.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:17:14.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:17:14.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:17:14.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:17:14.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:17:14.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:17:14.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-02-05T05:17:14.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:17:14.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:17:14.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89bcb514-eab6-440b-a6cd-23cae60b18d0 2024-02-05T05:17:14.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 89bcb514-eab6-440b-a6cd-23cae60b18d0' 2024-02-05T05:17:14.530 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 89bcb514-eab6-440b-a6cd-23cae60b18d0 2024-02-05T05:17:14.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:17:14.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBab8Blm0ftIBAAkFjDu1ZcKCvbhAwQl+6D6Q== 2024-02-05T05:17:14.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBab8Blm0ftIBAAkFjDu1ZcKCvbhAwQl+6D6Q=="}' 2024-02-05T05:17:14.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89bcb514-eab6-440b-a6cd-23cae60b18d0 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:17:14.805 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:17:14.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:17:14.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBab8Blm0ftIBAAkFjDu1ZcKCvbhAwQl+6D6Q== --osd-uuid 89bcb514-eab6-440b-a6cd-23cae60b18d0 2024-02-05T05:17:14.866 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:14.868+0000 7f9bc83cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:14.881 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:14.883+0000 7f9bc83cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:14.883 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:14.883+0000 7f9bc83cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:14.883 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:14.883+0000 7f9bc83cc640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:17:14.883 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:14.883+0000 7f9bc83cc640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:17:14.883 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:14.883+0000 7f9bc83cc640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:17:14.973 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:14.975+0000 7f9bc83cc640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:17:21.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:17:21.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:17:21.252 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:17:21.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:17:21.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:17:21.629 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:17:21.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:17:21.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T05:17:21.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:17:21.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:17:21.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:17:21.665 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:21.667+0000 7f089c47a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:21.697 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:21.699+0000 7f089c47a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:21.699 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:21.699+0000 7f089c47a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:21.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:17:21.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:17:21.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:17:21.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:17:21.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:17:21.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:21.964 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:17:21.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:17:21.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:21.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:17:22.255 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:22.255+0000 7f089c47a640 -1 Falling back to public interface 2024-02-05T05:17:22.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:23.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:23.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:23.282 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:17:23.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:17:23.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:23.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:17:23.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:24.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:24.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:24.591 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:17:24.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:17:24.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:24.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:17:24.824 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:24.823+0000 7f089c47a640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:17:24.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:25.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:25.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:25.938 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:17:25.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:17:25.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:25.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:17:26.244 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/450462124,v1:127.0.0.1:6803/450462124] [v2:127.0.0.1:6804/450462124,v1:127.0.0.1:6805/450462124] exists,up 89bcb514-eab6-440b-a6cd-23cae60b18d0 2024-02-05T05:17:26.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:17:26.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:17:26.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:17:26.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-02-05T05:17:26.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T05:17:26.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:17:26.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:17:26.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:17:26.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:17:26.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:17:26.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:17:26.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:17:26.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:17:26.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:17:26.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:17:26.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:17:26.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:17:26.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:17:26.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:17:26.249 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:17:26.249 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:17:26.249 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:17:26.249 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:17:26.249 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:26.249 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:17:26.249 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:17:26.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:17:26.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:17:26.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:17:26.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:17:26.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:17:26.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:17:26.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:17:26.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:17:26.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:17:26.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:17:26.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:17:26.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-02-05T05:17:26.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:17:26.251 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:17:26.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=04651e92-03a8-4541-b750-6030f7097c8b 2024-02-05T05:17:26.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 04651e92-03a8-4541-b750-6030f7097c8b' 2024-02-05T05:17:26.251 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 04651e92-03a8-4541-b750-6030f7097c8b 2024-02-05T05:17:26.252 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:17:26.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBmb8BlX2wqEBAABdvXuI+Jq9grylchobtpRw== 2024-02-05T05:17:26.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBmb8BlX2wqEBAABdvXuI+Jq9grylchobtpRw=="}' 2024-02-05T05:17:26.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 04651e92-03a8-4541-b750-6030f7097c8b -i td/osd-scrub-repair/1/new.json 2024-02-05T05:17:26.635 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:17:26.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:17:26.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBmb8BlX2wqEBAABdvXuI+Jq9grylchobtpRw== --osd-uuid 04651e92-03a8-4541-b750-6030f7097c8b 2024-02-05T05:17:26.687 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:26.687+0000 7ff35e5c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:26.728 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:26.727+0000 7ff35e5c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:26.730 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:26.731+0000 7ff35e5c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:26.730 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:26.731+0000 7ff35e5c0640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:17:26.730 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:26.731+0000 7ff35e5c0640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:17:26.730 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:26.731+0000 7ff35e5c0640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:17:26.807 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:26.807+0000 7ff35e5c0640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:17:33.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:17:33.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:17:33.698 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:17:33.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:17:33.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:17:34.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:17:34.083 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:17:34.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T05:17:34.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:17:34.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:17:34.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:17:34.117 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:34.119+0000 7f217ad96640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:34.144 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:34.143+0000 7f217ad96640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:34.145 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:34.147+0000 7f217ad96640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:34.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:17:34.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:17:34.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:17:34.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:17:34.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:17:34.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:34.415 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:17:34.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:17:34.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:34.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:17:34.706 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:34.707+0000 7f217ad96640 -1 Falling back to public interface 2024-02-05T05:17:34.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:35.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:35.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:35.730 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:17:35.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:17:35.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:35.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:17:36.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:37.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:37.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:37.038 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:17:37.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:17:37.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:37.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:17:37.353 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:37.355+0000 7f217ad96640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:17:37.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:38.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:38.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:38.358 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:17:38.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:17:38.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:38.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:17:38.697 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4000481998,v1:127.0.0.1:6811/4000481998] [v2:127.0.0.1:6812/4000481998,v1:127.0.0.1:6813/4000481998] exists,up 04651e92-03a8-4541-b750-6030f7097c8b 2024-02-05T05:17:38.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:17:38.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:17:38.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:17:38.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-02-05T05:17:38.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T05:17:38.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:17:38.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:17:38.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:17:38.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:17:38.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:17:38.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:17:38.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:17:38.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:17:38.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:17:38.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:17:38.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:17:38.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:17:38.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:17:38.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:17:38.700 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:17:38.700 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:17:38.700 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:17:38.700 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:17:38.700 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:38.700 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:17:38.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:17:38.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:17:38.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:17:38.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:17:38.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:17:38.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:17:38.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:17:38.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:17:38.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:17:38.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:17:38.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:17:38.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:17:38.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-02-05T05:17:38.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:17:38.703 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:17:38.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2c55c44e-a3c9-4005-a09a-0107ba0c6021 2024-02-05T05:17:38.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2c55c44e-a3c9-4005-a09a-0107ba0c6021' 2024-02-05T05:17:38.704 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 2c55c44e-a3c9-4005-a09a-0107ba0c6021 2024-02-05T05:17:38.704 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:17:38.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQByb8BlWO5nKxAAYgfCq2t9rAYx5LEypn0oZw== 2024-02-05T05:17:38.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQByb8BlWO5nKxAAYgfCq2t9rAYx5LEypn0oZw=="}' 2024-02-05T05:17:38.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2c55c44e-a3c9-4005-a09a-0107ba0c6021 -i td/osd-scrub-repair/2/new.json 2024-02-05T05:17:39.131 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:17:39.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:17:39.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQByb8BlWO5nKxAAYgfCq2t9rAYx5LEypn0oZw== --osd-uuid 2c55c44e-a3c9-4005-a09a-0107ba0c6021 2024-02-05T05:17:39.183 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:39.183+0000 7f72f3743640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:39.225 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:39.227+0000 7f72f3743640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:39.226 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:39.227+0000 7f72f3743640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:39.227 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:39.227+0000 7f72f3743640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:17:39.227 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:39.227+0000 7f72f3743640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:17:39.227 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:39.227+0000 7f72f3743640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:17:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:39.299+0000 7f72f3743640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:17:46.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:17:46.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:17:46.276 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T05:17:46.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:17:46.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:17:46.657 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T05:17:46.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:17:46.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:17:46.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:17:46.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:17:46.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T05:17:46.693 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:46.694+0000 7f617182b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:46.727 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:46.726+0000 7f617182b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:46.729 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:46.730+0000 7f617182b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:17:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:17:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:17:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:17:47.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:17:47.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:47.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:17:47.028 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:17:47.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:17:47.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:47.291 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:47.290+0000 7f617182b640 -1 Falling back to public interface 2024-02-05T05:17:47.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:48.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:48.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:48.373 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:17:48.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:17:48.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:48.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:17:48.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:49.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:49.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:49.687 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:17:49.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:17:49.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:49.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:17:49.997 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:17:49.998+0000 7f617182b640 -1 osd.2 0 log_to_monitors true 2024-02-05T05:17:50.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:51.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:51.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:51.009 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:17:51.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:17:51.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:51.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:17:51.340 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3849487451,v1:127.0.0.1:6819/3849487451] [v2:127.0.0.1:6820/3849487451,v1:127.0.0.1:6821/3849487451] exists,up 2c55c44e-a3c9-4005-a09a-0107ba0c6021 2024-02-05T05:17:51.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:17:51.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:17:51.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:17:51.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-02-05T05:17:51.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-02-05T05:17:51.850 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'testpool' created 2024-02-05T05:17:51.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:17:52.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-02-05T05:17:53.591 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 1 size to 2 2024-02-05T05:17:53.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-02-05T05:17:53.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:17:53.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:17:53.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:17:53.616 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:17:53.617 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:17:53.617 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:17:53.617 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:17:53.617 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:17:53.617 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:17:53.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:17:53.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:17:53.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:17:53.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:17:53.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:17:53.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:17:54.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:17:54.094 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:17:54.095 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:17:54.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:17:54.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:54.095 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:17:54.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-02-05T05:17:54.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-02-05T05:17:54.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-02-05T05:17:54.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:54.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:17:54.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-02-05T05:17:54.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-02-05T05:17:54.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2024-02-05T05:17:54.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:54.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:17:54.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-02-05T05:17:54.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-02-05T05:17:54.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509442' 2024-02-05T05:17:54.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:54.556 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-02-05T05:17:54.556 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:54.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:17:54.557 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-02-05T05:17:54.557 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:54.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-02-05T05:17:54.559 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836487 2024-02-05T05:17:54.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-02-05T05:17:54.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:17:54.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836487 2024-02-05T05:17:54.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:17:55.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:17:55.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:17:56.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836487 2024-02-05T05:17:56.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:56.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-02-05T05:17:56.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:56.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:17:56.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-02-05T05:17:56.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:56.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-02-05T05:17:56.192 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672965 2024-02-05T05:17:56.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-02-05T05:17:56.193 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:17:56.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672965 2024-02-05T05:17:56.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:56.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:56.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-02-05T05:17:56.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:17:56.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-02-05T05:17:56.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:56.537 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509442 2024-02-05T05:17:56.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-02-05T05:17:56.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-02-05T05:17:56.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:17:56.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-02-05T05:17:56.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:17:56.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:17:56.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:17:57.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:17:57.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:17:57.315 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:17:57.315 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:17:57.315 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:17:57.315 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:17:57.315 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:17:57.315 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:17:57.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:17:57.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:17:57.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:17:57.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:17:58.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:17:58.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:17:58.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:17:58.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-02-05T05:17:58.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-02-05T05:17:58.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:17:58.199 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-02-05T05:17:58.199 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:17:58.199 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:17:58.199 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T05:17:58.199 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:17:58.199 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:17:58.200 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:17:58.200 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:17:58.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:17:58.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:17:58.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:17:58.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:17:58.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:17:58.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:17:58.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:17:58.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:17:58.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:17:58.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:17:58.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:17:58.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:17:58.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:17:58.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:17:58.875 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:17:58.875 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:17:58.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:17:58.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:17:58.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:17:59.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:17:59.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:17:59.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:17:59.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:17:59.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:17:59.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:17:59.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:17:59.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-02-05T05:18:00.530 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-02-05T05:18:01.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T05:18:01.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:18:01.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:18:01.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:18:01.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:18:01.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:18:01.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:18:01.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:18:01.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:18:01.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:18:01.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:18:01.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:18:01.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:18:01.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:18:01.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:18:01.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:18:01.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:18:01.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:18:01.072 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:18:01.072 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:01.072 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:01.072 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:18:01.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:18:01.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:18:01.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:18:01.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:18:01.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:18:01.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:18:01.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:18:01.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:18:01.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:18:01.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-02-05T05:18:01.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:18:01.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:18:01.073 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:18:01.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T05:18:01.073 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:18:01.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:18:01.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:18:01.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:18:01.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:18:01.109 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:01.114+0000 7fe87da49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:01.112 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:01.114+0000 7fe87da49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:01.114 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:01.118+0000 7fe87da49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:01.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:18:01.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:18:01.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:18:01.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:18:01.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:18:01.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:01.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:18:01.384 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:18:01.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:01.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:01.643 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:01.646+0000 7fe87da49640 -1 Falling back to public interface 2024-02-05T05:18:01.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:02.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:02.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:02.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:18:02.716 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:18:02.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:02.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:03.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:04.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:04.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:04.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:18:04.027 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:18:04.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:04.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:04.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:04.412 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:04.414+0000 7fe87da49640 -1 osd.0 20 log_to_monitors true 2024-02-05T05:18:05.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:05.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:05.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:18:05.332 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:18:05.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:05.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:05.664 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/220992535,v1:127.0.0.1:6803/220992535] [v2:127.0.0.1:6804/220992535,v1:127.0.0.1:6805/220992535] exists,up 89bcb514-eab6-440b-a6cd-23cae60b18d0 2024-02-05T05:18:05.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:18:05.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:18:05.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:18:05.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:18:05.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:18:05.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:18:05.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:18:05.665 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:18:05.665 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:18:05.665 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:18:05.665 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:18:05.665 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:18:05.665 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:18:05.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:18:05.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:18:05.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:18:05.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:18:05.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:18:05.769 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:18:06.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:18:06.105 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:18:06.105 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:18:06.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:18:06.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:06.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:18:06.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-02-05T05:18:06.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-02-05T05:18:06.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-02-05T05:18:06.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:06.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:18:06.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-02-05T05:18:06.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-02-05T05:18:06.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672968' 2024-02-05T05:18:06.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:06.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:18:06.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-02-05T05:18:06.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-02-05T05:18:06.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672968 2-64424509446' 2024-02-05T05:18:06.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:06.552 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-02-05T05:18:06.552 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:06.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:18:06.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-02-05T05:18:06.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-02-05T05:18:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-02-05T05:18:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 103079215106 2024-02-05T05:18:06.556 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:06.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 103079215106 2024-02-05T05:18:06.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:18:07.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:18:07.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:08.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-02-05T05:18:08.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:08.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-02-05T05:18:08.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:08.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:18:08.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-02-05T05:18:08.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:08.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-02-05T05:18:08.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-02-05T05:18:08.187 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672968 2024-02-05T05:18:08.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:18:08.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-02-05T05:18:08.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:08.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-02-05T05:18:08.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:08.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:18:08.501 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-02-05T05:18:08.501 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:08.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-02-05T05:18:08.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-02-05T05:18:08.502 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509446 2024-02-05T05:18:08.503 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:18:08.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-02-05T05:18:08.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:18:08.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:08.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:09.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:18:09.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:18:09.170 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:18:09.170 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:18:09.170 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:18:09.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:18:09.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:18:09.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:18:09.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:18:09.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:18:09.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:09.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:09.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:18:09.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:18:09.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:18:09.880 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-02-05T05:18:09.880 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-02-05T05:18:09.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:18:09.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-02-05T05:18:09.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:18:10.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-02-05T05:18:10.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-02-05T05:18:10.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:18:10.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:18:10.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:18:10.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:18:10.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-02-05T05:18:10.546 INFO:tasks.workunit.client.0.smithi148.stdout:Affected PG 1.0 w/ primary 1 2024-02-05T05:18:10.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-02-05T05:18:10.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-02-05T05:18:10.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:18:10.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:18:10.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:18:10.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:18:10.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-02-05T05:17:51.728964+0000 2024-02-05T05:18:10.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-02-05T05:18:10.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-02-05T05:18:10.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-02-05T05:18:10.876 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-02-05T05:18:10.876 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:18:10.876 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:18:10.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:18:10.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:18:11.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-02-05T05:17:51.728964+0000 2024-02-05T05:18:11.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-02-05T05:18:11.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T05:18:11.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-02-05T05:18:11.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-02-05T05:18:11.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=3.0 2024-02-05T05:18:11.191 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-02-05T05:18:11.191 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T05:18:11.191 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T05:18:11.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T05:18:11.192 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:18:11.192 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:11.192 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:11.192 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.1.asok 2024-02-05T05:18:11.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61355/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-02-05T05:18:11.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T05:18:11.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-02-05T05:18:11.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T05:18:11.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-02-05T05:18:11.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_auto_repair 2024-02-05T05:18:11.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T05:18:11.313 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-02-05T05:18:11.314 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T05:18:11.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T05:18:11.314 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T05:18:11.314 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:18:11.314 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:11.314 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:11.314 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.1.asok 2024-02-05T05:18:11.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61355/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-02-05T05:18:11.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T05:18:11.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-02-05T05:18:11.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:18:11.421 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:18:11.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:18:11.715 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:18:11.715 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:18:11.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:18:11.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:11.716 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:18:11.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215108 2024-02-05T05:18:11.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215108 2024-02-05T05:18:11.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215108' 2024-02-05T05:18:11.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:11.857 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:18:11.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-02-05T05:18:11.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-02-05T05:18:11.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215108 1-42949672970' 2024-02-05T05:18:11.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:11.988 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:18:12.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-02-05T05:18:12.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-02-05T05:18:12.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215108 1-42949672970 2-64424509448' 2024-02-05T05:18:12.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:12.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215108 2024-02-05T05:18:12.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:12.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:18:12.132 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215108 2024-02-05T05:18:12.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:12.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215108 2024-02-05T05:18:12.134 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 103079215108 2024-02-05T05:18:12.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2024-02-05T05:18:12.134 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:12.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215107 -lt 103079215108 2024-02-05T05:18:12.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:18:13.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:18:13.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:13.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215108 -lt 103079215108 2024-02-05T05:18:13.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:13.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-02-05T05:18:13.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:13.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:18:13.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:13.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-02-05T05:18:13.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-02-05T05:18:13.734 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672970 2024-02-05T05:18:13.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-02-05T05:18:13.735 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:18:14.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-02-05T05:18:14.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:14.025 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-02-05T05:18:14.025 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:14.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:18:14.028 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-02-05T05:18:14.028 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:14.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-02-05T05:18:14.030 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509448 2024-02-05T05:18:14.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-02-05T05:18:14.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:18:14.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-02-05T05:18:14.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-02-05T05:18:14.351 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05 05:18:14.356927935+00:00 2024-02-05T05:18:14.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-02-05T05:18:14.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T05:18:14.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T05:18:14.354 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:18:14.354 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:14.354 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:14.354 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.1.asok 2024-02-05T05:18:14.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61355/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-02-05T05:18:14.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-02-05T05:18:14.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-02-05T05:18:14.477 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-02-05T05:18:14.477 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-02-05T05:18:14.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-02-05T05:18:14.625 INFO:tasks.workunit.client.0.smithi148.stdout:{"deep":true,"must":false,"stamp":"2024-01-22T05:16:34.468379+0000"}0 ) state now: "active+clean" 2024-02-05T05:18:14.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-02-05T05:18:14.626 INFO:tasks.workunit.client.0.smithi148.stdout:"active+clean" 2024-02-05T05:18:14.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-02-05T05:18:14.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-02-05T05:18:14.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-02-05T05:18:14.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-02-05T05:18:14.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-02-05T05:18:14.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-02-05T05:18:14.927 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-02-05T05:18:14.927 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-02-05T05:18:15.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-02-05T05:18:15.076 INFO:tasks.workunit.client.0.smithi148.stdout:1 ) state now: "active+clean" 2024-02-05T05:18:15.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-02-05T05:18:15.077 INFO:tasks.workunit.client.0.smithi148.stdout:"active+clean" 2024-02-05T05:18:15.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-02-05T05:18:15.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-02-05T05:18:15.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-02-05T05:18:15.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-02-05T05:18:15.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-02-05T05:18:15.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-02-05T05:18:15.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-02-05T05:18:15.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-02-05T05:18:15.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-02-05T05:18:15.555 INFO:tasks.workunit.client.0.smithi148.stdout:2 ) state now: "active+clean+scrubbing+deep" 2024-02-05T05:18:15.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean+scrubbing+deep"' 2024-02-05T05:18:15.555 INFO:tasks.workunit.client.0.smithi148.stdout:found scrub 2024-02-05T05:18:15.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-02-05T05:18:15.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-02-05T05:18:15.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-02-05T05:18:15.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-02-05T05:18:15.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-02-05T05:18:15.556 INFO:tasks.workunit.client.0.smithi148.stdout:initiate_and_fetch_state ret: 0 2024-02-05T05:18:15.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-02-05T05:18:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T05:18:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=td/osd-scrub-repair 2024-02-05T05:18:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-02-05T05:18:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=0 2024-02-05T05:18:15.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T05:18:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.td/osd-scrub-repair 2024-02-05T05:18:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.td/osd-scrub-repair 2024-02-05T05:18:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-02-05T05:18:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:18:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:15.558 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.td/osd-scrub-repair.asok 2024-02-05T05:18:15.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61355/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-02-05T05:18:15.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test == true 2024-02-05T05:18:15.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1168: test: ==: unary operator expected 2024-02-05T05:18:15.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-02-05T05:18:15.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-02-05T05:18:15.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-02-05T05:18:15.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-02-05T05:18:15.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-02-05T05:18:15.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-02-05T05:18:15.643 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-02-05T05:18:15.643 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-02-05T05:18:15.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-02-05T05:18:15.784 INFO:tasks.workunit.client.0.smithi148.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-02-05T05:18:15.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-02-05T05:18:15.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-02-05T05:18:15.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-02-05T05:18:15.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-02-05T05:18:16.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-02-05T05:18:16.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-02-05T05:18:16.086 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-02-05T05:18:16.086 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-02-05T05:18:16.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-02-05T05:18:16.240 INFO:tasks.workunit.client.0.smithi148.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-02-05T05:18:16.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-02-05T05:18:16.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-02-05T05:18:16.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-02-05T05:18:16.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-02-05T05:18:16.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-02-05T05:18:16.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-02-05T05:18:16.542 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-02-05T05:18:16.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-02-05T05:18:16.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-02-05T05:18:16.700 INFO:tasks.workunit.client.0.smithi148.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-02-05T05:18:16.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-02-05T05:18:16.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-02-05T05:18:16.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-02-05T05:18:16.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-02-05T05:18:17.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-02-05T05:18:17.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-02-05T05:18:17.003 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-02-05T05:18:17.003 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-02-05T05:18:17.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-02-05T05:18:17.161 INFO:tasks.workunit.client.0.smithi148.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-02-05T05:18:17.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-02-05T05:18:17.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-02-05T05:18:17.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-02-05T05:18:17.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-02-05T05:18:17.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-02-05T05:18:17.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-02-05T05:18:17.464 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-02-05T05:18:17.464 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-02-05T05:18:17.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-02-05T05:18:17.658 INFO:tasks.workunit.client.0.smithi148.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-02-05T05:18:17.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-02-05T05:18:17.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-02-05T05:18:17.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-02-05T05:18:17.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-02-05T05:18:17.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:18:17.659 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:18:17.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:18:17.964 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:18:17.964 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:18:17.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:18:17.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:17.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:18:18.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215110 2024-02-05T05:18:18.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215110 2024-02-05T05:18:18.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110' 2024-02-05T05:18:18.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:18.129 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:18:18.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-02-05T05:18:18.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-02-05T05:18:18.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110 1-42949672972' 2024-02-05T05:18:18.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:18.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:18:18.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-02-05T05:18:18.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-02-05T05:18:18.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110 1-42949672972 2-64424509450' 2024-02-05T05:18:18.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:18.420 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215110 2024-02-05T05:18:18.420 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:18.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:18:18.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215110 2024-02-05T05:18:18.423 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:18.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215110 2024-02-05T05:18:18.424 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 103079215110 2024-02-05T05:18:18.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215110' 2024-02-05T05:18:18.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:18.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215110 2024-02-05T05:18:18.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:18:19.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:18:19.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:20.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215110 -lt 103079215110 2024-02-05T05:18:20.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:20.132 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-02-05T05:18:20.132 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:20.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:18:20.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-02-05T05:18:20.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:20.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-02-05T05:18:20.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-02-05T05:18:20.137 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672972 2024-02-05T05:18:20.137 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:18:20.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-02-05T05:18:20.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:20.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-02-05T05:18:20.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:20.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:18:20.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-02-05T05:18:20.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:20.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-02-05T05:18:20.470 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509450 2024-02-05T05:18:20.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-02-05T05:18:20.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:18:20.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-02-05T05:18:20.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-02-05T05:18:21.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-02-05T05:18:21.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-02-05T05:18:21.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-02-05T05:18:21.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-02-05T05:18:21.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-02-05T05:18:21.254 INFO:tasks.workunit.client.0.smithi148.stdout:wait-scrub-end state now: "active+clean" 2024-02-05T05:18:21.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-02-05T05:18:21.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-02-05T05:18:21.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-02-05T05:18:21.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-02-05T05:18:21.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-02-05T05:18:21.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-02-05T05:18:21.541 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T05:18:21.542 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-02-05T05:18:18.733123+0000 20'1 25:58 [1,0] 1 [1,0] 1 20'1 2024-02-05T05:18:18.362758+0000 20'1 2024-02-05T05:18:18.362758+0000 0 4 periodic scrub scheduled @ 2024-02-06T05:18:18.362758+0000 1 0 2024-02-05T05:18:21.542 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T05:18:21.542 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T05:18:21.542 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T05:18:21.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-02-05T05:18:21.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:18:21.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:18:21.561 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T05:18:21.561 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:18:21.561 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:18:21.562 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:18:21.562 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:18:21.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:18:21.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:18:21.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:18:22.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T05:18:22.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:18:22.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:18:22.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:18:22.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:18:22.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T05:18:22.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:18:22.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:18:22.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:18:22.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:18:22.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:18:22.232 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:18:22.232 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:18:22.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:18:22.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:18:22.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:18:22.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:18:22.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T05:18:22.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:18:22.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:18:22.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:18:22.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:18:22.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:18:22.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-02-05T05:18:24.040 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:18:24.041 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:18:24.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T05:18:24.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:18:24.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:18:24.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:18:24.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:18:24.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:18:24.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:18:24.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:18:24.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:18:24.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:18:24.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:18:24.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:18:24.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:18:24.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:18:24.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:18:24.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:18:24.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:18:24.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:18:24.337 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:18:24.337 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:24.337 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:24.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:18:24.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:18:24.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:18:24.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:18:24.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:18:24.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:18:24.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:18:24.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:18:24.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:18:24.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:18:24.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-02-05T05:18:24.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:18:24.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:18:24.339 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:18:24.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T05:18:24.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:18:24.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:18:24.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:18:24.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:18:24.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:18:24.374 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:24.377+0000 7f65ef869640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:24.377 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:24.377+0000 7f65ef869640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:24.378 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:24.381+0000 7f65ef869640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:24.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:18:24.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:18:24.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:18:24.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:18:24.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:18:24.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:24.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:18:24.657 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:18:24.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:24.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:24.920 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:24.921+0000 7f65ef869640 -1 Falling back to public interface 2024-02-05T05:18:25.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:26.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:26.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:26.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:18:26.014 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:18:26.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:26.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:26.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:27.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:27.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:27.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:18:27.337 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:18:27.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:27.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:27.516 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:27.517+0000 7f65ef869640 -1 osd.0 25 log_to_monitors true 2024-02-05T05:18:27.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:18:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:18:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/3500566502,v1:127.0.0.1:6803/3500566502] [v2:127.0.0.1:6804/3500566502,v1:127.0.0.1:6805/3500566502] exists,up 89bcb514-eab6-440b-a6cd-23cae60b18d0 2024-02-05T05:18:29.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:18:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:18:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:18:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:18:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:18:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:18:29.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:18:29.034 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:18:29.035 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:18:29.035 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:18:29.035 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:18:29.035 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:18:29.035 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:18:29.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:18:29.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:18:29.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:18:29.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:18:29.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:18:29.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:18:29.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:18:29.463 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:18:29.463 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:18:29.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:18:29.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:29.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:18:29.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-02-05T05:18:29.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-02-05T05:18:29.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-02-05T05:18:29.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:29.645 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:18:29.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-02-05T05:18:29.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-02-05T05:18:29.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672976' 2024-02-05T05:18:29.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:29.826 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:18:29.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-02-05T05:18:29.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-02-05T05:18:29.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672976 2-64424509453' 2024-02-05T05:18:29.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:29.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-02-05T05:18:29.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:29.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:18:29.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-02-05T05:18:29.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:29.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-02-05T05:18:29.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-02-05T05:18:29.973 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 124554051586 2024-02-05T05:18:29.974 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:30.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-02-05T05:18:30.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:18:31.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:18:31.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:31.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-02-05T05:18:31.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:31.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:31.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-02-05T05:18:31.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:18:31.587 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-02-05T05:18:31.587 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:31.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-02-05T05:18:31.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-02-05T05:18:31.589 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672976 2024-02-05T05:18:31.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:18:31.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-02-05T05:18:31.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:31.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-02-05T05:18:31.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:31.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:18:31.897 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-02-05T05:18:31.897 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:31.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-02-05T05:18:31.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-02-05T05:18:31.899 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509453 2024-02-05T05:18:31.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:18:32.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-02-05T05:18:32.212 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:18:32.212 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:32.212 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:32.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:18:32.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:18:32.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:18:32.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:18:32.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:18:32.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:18:32.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:18:32.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:18:32.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:18:32.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:18:32.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:32.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:33.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:18:33.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:18:33.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:18:33.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-02-05T05:18:33.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:18:33.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:18:33.282 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T05:18:33.283 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:18:33.283 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:18:33.283 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:18:33.283 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:18:33.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:18:33.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T05:18:33.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:18:33.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T05:18:33.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:18:33.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:18:33.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:18:33.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:18:33.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T05:18:33.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:18:33.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:18:33.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:18:33.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:18:33.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:18:33.967 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:18:33.967 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:18:33.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:18:33.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:18:33.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:18:34.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:18:34.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T05:18:34.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:18:34.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:18:34.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:18:34.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:18:34.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:18:34.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T05:18:36.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T05:18:36.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:18:36.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:18:36.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:18:36.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:18:36.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:18:36.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:18:36.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:18:36.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:18:36.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:18:36.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:18:36.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:18:36.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:18:36.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:18:36.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:18:36.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:18:36.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:18:36.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:18:36.247 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:18:36.247 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:36.247 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:36.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:18:36.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:18:36.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:18:36.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:18:36.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:18:36.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:18:36.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:18:36.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:18:36.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:18:36.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:18:36.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-02-05T05:18:36.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:18:36.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:18:36.249 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:18:36.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T05:18:36.249 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:18:36.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:18:36.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:18:36.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:18:36.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:18:36.286 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:36.288+0000 7f66795ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:36.288 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:36.288+0000 7f66795ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:36.290 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:36.292+0000 7f66795ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:36.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:18:36.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:18:36.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:18:36.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:18:36.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:18:36.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:36.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:18:36.571 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:18:36.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:36.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:36.824 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:36.824+0000 7f66795ca640 -1 Falling back to public interface 2024-02-05T05:18:36.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:37.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:37.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:37.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:18:37.912 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:18:37.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:37.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:38.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:39.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:39.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:39.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:18:39.229 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:18:39.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:39.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:39.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:39.557 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:39.556+0000 7f66795ca640 -1 osd.0 30 log_to_monitors true 2024-02-05T05:18:40.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:40.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:40.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:18:40.547 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:18:40.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:40.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:40.889 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 34 up_thru 0 down_at 31 last_clean_interval [29,30) [v2:127.0.0.1:6802/4102567275,v1:127.0.0.1:6803/4102567275] [v2:127.0.0.1:6804/4102567275,v1:127.0.0.1:6805/4102567275] exists,up 89bcb514-eab6-440b-a6cd-23cae60b18d0 2024-02-05T05:18:40.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:18:40.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:18:40.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:18:40.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:18:40.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:18:40.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:18:40.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:18:40.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:18:40.891 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:18:40.891 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:18:40.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:18:40.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:18:40.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:18:40.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:18:40.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:18:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:18:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:18:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:18:40.992 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:18:41.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:18:41.396 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:18:41.396 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:18:41.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:18:41.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:41.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:18:41.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-02-05T05:18:41.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-02-05T05:18:41.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066' 2024-02-05T05:18:41.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:41.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:18:41.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-02-05T05:18:41.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-02-05T05:18:41.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-42949672979' 2024-02-05T05:18:41.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:41.725 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:18:41.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-02-05T05:18:41.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-02-05T05:18:41.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-42949672979 2-64424509457' 2024-02-05T05:18:41.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:41.865 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888066 2024-02-05T05:18:41.865 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:41.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:18:41.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888066 2024-02-05T05:18:41.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:41.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-02-05T05:18:41.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-02-05T05:18:41.868 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 146028888066 2024-02-05T05:18:41.868 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:42.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 146028888066 2024-02-05T05:18:42.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:18:43.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:18:43.170 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:43.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888066 2024-02-05T05:18:43.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:43.490 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-02-05T05:18:43.491 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:43.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:18:43.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-02-05T05:18:43.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:43.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-02-05T05:18:43.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-02-05T05:18:43.494 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672979 2024-02-05T05:18:43.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:18:43.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-02-05T05:18:43.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:43.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-02-05T05:18:43.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:43.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:18:43.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-02-05T05:18:43.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:43.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-02-05T05:18:43.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-02-05T05:18:43.824 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509457 2024-02-05T05:18:43.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:18:44.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-02-05T05:18:44.120 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:18:44.120 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:44.120 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:44.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:18:44.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:18:44.510 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:18:44.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:18:44.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:18:44.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:18:44.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:18:44.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:18:44.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:18:44.838 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:18:44.838 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:44.838 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:45.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:18:45.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:18:45.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:18:45.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:18:45.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-02-05T05:18:45.215 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:18:18.361+0000 7f2161525640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T05:18:45.215 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:18:18.361+0000 7f2161525640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-02-05T05:18:45.215 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:18:18.361+0000 7f2161525640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-02-05T05:18:45.215 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:18:18.361+0000 7f2161525640 20 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-02-05T05:18:45.215 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T05:18:18.361+0000 7f2161525640 19 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T05:18:45.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:18:45.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:18:45.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:18:45.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:18:45.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:18:45.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:18:45.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:18:45.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:18:45.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:18:45.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:18:45.526 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:18:45.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:18:45.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:18:45.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:18:45.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:18:45.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:18:45.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:18:45.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:18:45.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:18:45.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:18:45.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:18:45.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:18:45.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:18:45.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:18:45.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:45.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:45.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:18:45.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:18:45.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:18:45.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:18:45.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:18:45.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:18:45.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:18:45.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:18:45.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:18:45.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:18:45.549 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:18:45.550 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:18:45.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:18:45.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:18:45.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:18:45.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:18:45.552 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:18:45.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:18:45.553 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:18:45.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:18:45.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:18:45.555 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:18:45.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:18:45.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:18:45.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:18:45.557 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:18:45.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:18:45.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:18:45.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:18:45.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:18:45.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:45.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:45.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:18:45.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:18:45.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:18:45.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:18:45.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:18:45.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:45.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:45.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T05:18:45.565 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:18:45.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:18:45.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:18:45.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:18:45.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-02-05T05:18:45.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-02-05T05:18:45.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T05:18:45.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-02-05T05:18:45.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-02-05T05:18:45.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-02-05T05:18:45.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:18:45.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:18:45.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:18:45.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:18:45.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:18:45.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:18:45.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:18:45.973 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:18:45.973 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:18:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:18:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:45.973 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:18:45.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:18:46.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:18:46.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:18:46.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:18:46.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:18:46.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:18:46.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:18:46.185 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:18:46.185 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:18:46.185 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:18:46.185 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:18:46.185 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:46.186 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:46.186 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:18:46.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:18:46.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T05:18:46.301 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:18:46.301 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:18:46.301 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:18:46.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:18:46.303 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:18:46.303 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:18:46.303 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:18:46.303 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:18:46.303 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:18:46.303 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:46.303 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:46.303 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:18:46.303 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:18:46.304 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T05:18:46.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-02-05T05:18:46.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:18:46.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:18:46.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:18:46.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:18:46.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:18:46.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:18:47.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:18:47.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:18:47.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:18:47.283 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:18:47.283 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:47.283 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:47.284 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:18:47.284 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:18:47.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:18:47.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:18:47.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-02-05T05:18:47.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:18:47.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:18:47.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:18:47.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:18:47.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:18:47.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:18:47.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:18:47.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:18:47.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:18:47.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:18:47.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:18:47.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:18:47.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:18:47.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:18:47.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:18:47.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:18:47.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:18:47.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:18:47.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:18:47.327 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:18:47.327 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:47.327 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:18:47.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:18:47.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:18:47.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:18:47.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:18:47.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:18:47.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:18:47.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:18:47.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:18:47.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:18:47.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:18:47.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:18:47.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:18:47.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:18:47.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:18:47.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:18:47.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=01a81c63-908f-442b-a73b-decf7637cc5f 2024-02-05T05:18:47.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 01a81c63-908f-442b-a73b-decf7637cc5f' 2024-02-05T05:18:47.328 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 01a81c63-908f-442b-a73b-decf7637cc5f 2024-02-05T05:18:47.329 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:18:47.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3b8BlAw2oFBAAg401+JEgMs0NVw9yI4elQw== 2024-02-05T05:18:47.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3b8BlAw2oFBAAg401+JEgMs0NVw9yI4elQw=="}' 2024-02-05T05:18:47.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 01a81c63-908f-442b-a73b-decf7637cc5f -i td/osd-scrub-repair/0/new.json 2024-02-05T05:18:47.578 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:18:47.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:18:47.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQC3b8BlAw2oFBAAg401+JEgMs0NVw9yI4elQw== --osd-uuid 01a81c63-908f-442b-a73b-decf7637cc5f 2024-02-05T05:18:47.638 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:47.640+0000 7f0ebb278640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:47.653 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:47.656+0000 7f0ebb278640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:47.655 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:47.656+0000 7f0ebb278640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:47.655 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:47.656+0000 7f0ebb278640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:18:47.655 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:47.656+0000 7f0ebb278640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:18:47.655 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:47.656+0000 7f0ebb278640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:18:47.746 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:47.744+0000 7f0ebb278640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:18:53.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:18:53.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:18:53.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:18:53.744 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:18:53.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:18:54.173 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:18:54.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:18:54.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:18:54.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:18:54.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:18:54.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:18:54.209 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:54.208+0000 7fc0ff094640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:54.244 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:54.244+0000 7fc0ff094640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:54.245 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:54.244+0000 7fc0ff094640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:54.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:18:54.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:18:54.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:18:54.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:18:54.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:18:54.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:54.517 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:18:54.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:18:54.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:54.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:54.816 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:54.816+0000 7fc0ff094640 -1 Falling back to public interface 2024-02-05T05:18:54.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:55.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:55.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:55.859 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:18:55.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:18:55.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:55.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:56.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:57.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:57.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:57.176 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:18:57.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:18:57.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:57.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:57.440 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:18:57.440+0000 7fc0ff094640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:18:57.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:58.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:58.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:58.489 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:18:58.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:18:58.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:58.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:18:58.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:59.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:59.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:59.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:18:59.806 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:18:59.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:59.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:00.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:01.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:01.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:01.131 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:19:01.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:19:01.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:01.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:01.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:02.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:02.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:02.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:19:02.627 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:19:02.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:02.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:02.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:03.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:03.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:03.943 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:19:03.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:19:03.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:03.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:04.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:05.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:05.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:05.352 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:19:05.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:19:05.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:05.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:05.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:06.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:06.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:06.671 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:19:06.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:19:06.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:06.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:06.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:07.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:07.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:07.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:19:07.999 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:19:08.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:08.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:08.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:09.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:09.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:09.353 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:19:09.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:19:09.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:09.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:09.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:10.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:10.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:10.672 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:19:10.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:19:10.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:10.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:10.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:11.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:11.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:11.980 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:19:11.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:19:11.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:11.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:12.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:13.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:13.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:13.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:19:13.411 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:19:13.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:13.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:14.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:15.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:15.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:15.156 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:19:15.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:19:15.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:15.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:15.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:16.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:16.757 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:19:16.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:16.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:19:16.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:16.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:17.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:18.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:18.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:18.513 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:19:18.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:19:18.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:18.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:18.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:19.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:19.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:19.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:19:19.936 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:19:19.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:19.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:20.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:21.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:21.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:21.425 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:19:21.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:19:21.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:21.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:22.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:23.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:23.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:23.240 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:19:23.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:19:23.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:23.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:23.554 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2841187658,v1:127.0.0.1:6803/2841187658] [v2:127.0.0.1:6804/2841187658,v1:127.0.0.1:6805/2841187658] exists,up 01a81c63-908f-442b-a73b-decf7637cc5f 2024-02-05T05:19:23.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:19:23.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:19:23.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:19:23.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:19:23.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:19:23.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:19:23.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:19:23.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:19:23.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:19:23.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:19:23.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:19:23.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:19:23.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:19:23.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:19:23.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:19:23.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:19:23.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:19:23.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:19:23.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:19:23.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:19:23.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:19:23.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:19:23.561 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:19:23.561 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:19:23.561 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:19:23.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:19:23.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:19:23.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:19:23.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:19:23.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:19:23.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:19:23.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:19:23.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:19:23.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:19:23.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:19:23.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:19:23.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:19:23.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:19:23.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:19:23.562 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:19:23.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9430a457-82bb-43d8-85de-e1cd0b008c61 2024-02-05T05:19:23.563 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 9430a457-82bb-43d8-85de-e1cd0b008c61 2024-02-05T05:19:23.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9430a457-82bb-43d8-85de-e1cd0b008c61' 2024-02-05T05:19:23.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:19:23.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbb8BleQKaIhAAV46gLaE5W5EaVLWtLF9GzA== 2024-02-05T05:19:23.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbb8BleQKaIhAAV46gLaE5W5EaVLWtLF9GzA=="}' 2024-02-05T05:19:23.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9430a457-82bb-43d8-85de-e1cd0b008c61 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:19:23.954 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:19:23.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:19:23.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDbb8BleQKaIhAAV46gLaE5W5EaVLWtLF9GzA== --osd-uuid 9430a457-82bb-43d8-85de-e1cd0b008c61 2024-02-05T05:19:24.012 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:19:24.011+0000 7fc4052d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:24.056 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:19:24.055+0000 7fc4052d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:24.058 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:19:24.055+0000 7fc4052d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:24.058 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:19:24.059+0000 7fc4052d5640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:19:24.058 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:19:24.059+0000 7fc4052d5640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:19:24.058 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:19:24.059+0000 7fc4052d5640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:19:24.139 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:19:24.143+0000 7fc4052d5640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:19:30.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:19:30.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:19:30.879 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:19:30.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:19:30.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:19:31.268 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:19:31.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:19:31.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:19:31.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:19:31.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:19:31.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:19:31.305 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:19:31.302+0000 7f9378734640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:31.338 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:19:31.338+0000 7f9378734640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:31.340 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:19:31.338+0000 7f9378734640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:19:31.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:19:31.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:19:31.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:19:31.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:19:31.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:31.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:19:31.552 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:19:31.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:31.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:31.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:31.916 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:19:31.914+0000 7f9378734640 -1 Falling back to public interface 2024-02-05T05:19:32.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:32.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:32.882 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:19:32.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:19:32.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:32.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:33.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:34.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:34.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:34.190 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:19:34.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:19:34.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:34.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:34.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:34.680 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:19:34.678+0000 7f9378734640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:19:35.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:35.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:35.510 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:19:35.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:19:35.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:35.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:35.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:36.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:36.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:36.866 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:19:36.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:19:36.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:36.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:37.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:38.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:38.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:38.188 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:19:38.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:19:38.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:38.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:38.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:39.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:39.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:39.505 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:19:39.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:19:39.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:39.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:39.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:40.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:40.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:40.945 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:19:40.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:19:40.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:40.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:41.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:42.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:42.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:42.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:19:42.265 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:19:42.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:42.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:42.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:43.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:43.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:43.585 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:19:43.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:19:43.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:43.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:43.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:44.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:44.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:44.903 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:19:44.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:19:44.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:44.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:45.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:46.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:46.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:46.286 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:19:46.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:19:46.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:46.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:46.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:47.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:47.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:47.735 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:19:47.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:19:47.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:47.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:48.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:49.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:49.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:49.083 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:19:49.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:19:49.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:49.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:49.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:50.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:50.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:50.675 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:19:50.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:19:50.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:50.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:51.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:52.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:52.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:52.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:19:52.504 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:19:52.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:52.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:52.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:53.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:53.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:53.869 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:19:53.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:19:53.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:53.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:54.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:55.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:55.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:55.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:19:55.671 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:19:55.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:55.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:56.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:57.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:57.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:57.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:19:57.209 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:19:57.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:57.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:58.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:59.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:59.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:59.166 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:19:59.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:19:59.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:59.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:59.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:00.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:00.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:00.878 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:20:00.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:20:00.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:00.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:01.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:02.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:02.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:02.202 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T05:20:02.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T05:20:02.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:02.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:02.518 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/121758528,v1:127.0.0.1:6811/121758528] [v2:127.0.0.1:6812/121758528,v1:127.0.0.1:6813/121758528] exists,up 9430a457-82bb-43d8-85de-e1cd0b008c61 2024-02-05T05:20:02.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:20:02.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:20:02.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:20:02.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:20:02.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:20:02.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:20:02.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:20:02.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:20:02.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:20:02.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:20:02.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:20:02.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:20:02.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:20:02.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:20:02.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:20:02.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:20:02.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:20:02.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:20:02.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:20:02.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:20:02.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:20:02.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:20:02.525 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:20:02.525 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:20:02.525 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:20:02.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:20:02.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:20:02.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:20:02.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:20:02.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:20:02.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:20:02.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:20:02.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:20:02.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:20:02.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:20:02.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:20:02.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:20:02.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:20:02.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:20:02.526 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:20:02.527 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 de1598a8-2043-4df9-8513-e5f695f88bfd 2024-02-05T05:20:02.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=de1598a8-2043-4df9-8513-e5f695f88bfd 2024-02-05T05:20:02.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 de1598a8-2043-4df9-8513-e5f695f88bfd' 2024-02-05T05:20:02.527 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:20:02.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACcMBlL2KmIBAAV53VlKYcko3RJlvKtYmWhw== 2024-02-05T05:20:02.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACcMBlL2KmIBAAV53VlKYcko3RJlvKtYmWhw=="}' 2024-02-05T05:20:02.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new de1598a8-2043-4df9-8513-e5f695f88bfd -i td/osd-scrub-repair/2/new.json 2024-02-05T05:20:02.905 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:20:02.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:20:02.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQACcMBlL2KmIBAAV53VlKYcko3RJlvKtYmWhw== --osd-uuid de1598a8-2043-4df9-8513-e5f695f88bfd 2024-02-05T05:20:02.964 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:20:02.965+0000 7fbd2de60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:03.007 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:20:03.009+0000 7fbd2de60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:03.008 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:20:03.009+0000 7fbd2de60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:03.009 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:20:03.009+0000 7fbd2de60640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:20:03.010 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:20:03.009+0000 7fbd2de60640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:20:03.010 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:20:03.009+0000 7fbd2de60640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:20:03.092 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:20:03.093+0000 7fbd2de60640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:20:09.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:20:09.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:20:09.957 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T05:20:09.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:20:09.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:20:10.351 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T05:20:10.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:20:10.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:20:10.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:20:10.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:20:10.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:20:10.390 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:20:10.389+0000 7fa7f9a53640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:10.418 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:20:10.417+0000 7fa7f9a53640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:10.419 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:20:10.421+0000 7fa7f9a53640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:10.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:20:10.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:20:10.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:20:10.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:20:10.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:20:10.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:10.680 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:20:10.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:20:10.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:10.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:10.965 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:20:10.965+0000 7fa7f9a53640 -1 Falling back to public interface 2024-02-05T05:20:10.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:11.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:11.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:11.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:20:11.989 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:20:11.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:11.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:12.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:13.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:13.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:13.308 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:20:13.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:20:13.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:13.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:13.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:13.860 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:20:13.861+0000 7fa7f9a53640 -1 osd.2 0 log_to_monitors true 2024-02-05T05:20:14.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:14.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:14.618 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:20:14.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:20:14.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:14.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:15.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:16.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:16.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:16.144 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:20:16.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:20:16.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:16.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:16.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:17.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:17.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:17.489 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:20:17.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:20:17.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:17.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:17.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:18.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:18.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:18.891 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:20:18.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:20:18.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:18.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:19.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:20.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:20.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:20.225 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:20:20.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:20:20.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:20.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:20.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:21.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:21.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:21.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:20:21.544 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:20:21.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:21.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:22.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:23.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:23.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:23.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:20:23.048 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:20:23.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:23.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:23.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:24.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:24.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:24.551 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:20:24.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:20:24.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:24.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:24.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:25.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:25.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:25.890 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:20:25.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:20:25.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:25.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:26.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:27.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:27.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:27.192 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:20:27.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:20:27.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:27.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:27.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:28.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:28.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:28.514 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:20:28.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:20:28.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:28.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:28.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:29.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:29.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:29.885 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:20:29.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:20:29.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:29.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:30.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:31.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:31.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:31.208 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:20:31.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:20:31.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:31.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:31.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:32.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:32.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:32.785 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:20:32.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:20:32.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:32.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:33.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:34.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:34.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:34.599 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:20:34.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:20:34.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:34.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:34.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:35.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:35.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:35.953 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:20:35.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:20:35.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:35.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:36.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:37.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:37.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:37.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:20:37.276 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:20:37.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:37.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:37.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:38.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:38.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:38.953 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:20:38.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:20:38.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:38.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:39.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:40.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:40.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:40.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T05:20:40.543 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T05:20:40.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:40.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:40.909 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2124973846,v1:127.0.0.1:6819/2124973846] [v2:127.0.0.1:6820/2124973846,v1:127.0.0.1:6821/2124973846] exists,up de1598a8-2043-4df9-8513-e5f695f88bfd 2024-02-05T05:20:40.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:20:40.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:20:40.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:20:40.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-02-05T05:20:40.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:20:41.224 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T05:20:41.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:20:41.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:20:41.668 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T05:20:41.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:20:42.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:20:45.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-02-05T05:20:45.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:20:45.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:20:45.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:20:45.102 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:20:45.103 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:20:45.103 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:20:45.103 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:20:45.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:20:45.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:20:45.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:20:45.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:20:45.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:20:45.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:20:45.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:20:45.232 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:20:45.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:20:45.566 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:20:45.566 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:20:45.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:20:45.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:20:45.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:20:45.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-02-05T05:20:45.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-02-05T05:20:45.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-02-05T05:20:45.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:20:45.717 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:20:45.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476754 2024-02-05T05:20:45.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476754 2024-02-05T05:20:45.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-68719476754' 2024-02-05T05:20:45.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:20:45.894 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:20:46.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855364 2024-02-05T05:20:46.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855364 2024-02-05T05:20:46.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-68719476754 2-150323855364' 2024-02-05T05:20:46.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:20:46.038 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-02-05T05:20:46.038 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:20:46.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:20:46.039 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-02-05T05:20:46.039 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:20:46.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-02-05T05:20:46.040 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836502 2024-02-05T05:20:46.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-02-05T05:20:46.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:20:46.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-02-05T05:20:46.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:20:46.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476754 2024-02-05T05:20:46.334 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:20:46.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:20:46.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476754 2024-02-05T05:20:46.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:20:46.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476754 2024-02-05T05:20:46.338 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 68719476754 2024-02-05T05:20:46.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476754' 2024-02-05T05:20:46.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:20:46.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476753 -lt 68719476754 2024-02-05T05:20:46.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:20:47.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:20:47.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:20:48.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476755 -lt 68719476754 2024-02-05T05:20:48.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:20:48.040 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-150323855364 2024-02-05T05:20:48.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:20:48.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:20:48.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-150323855364 2024-02-05T05:20:48.043 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:20:48.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855364 2024-02-05T05:20:48.044 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 150323855364 2024-02-05T05:20:48.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 150323855364' 2024-02-05T05:20:48.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:20:48.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855364 2024-02-05T05:20:48.376 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:20:48.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:20:48.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:20:48.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:20:48.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:20:48.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:20:48.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:20:48.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:20:48.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:20:48.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:20:48.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:20:49.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:20:49.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:20:49.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:20:49.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:20:49.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:20:49.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:20:49.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:20:49.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-02-05T05:20:49.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:20:49.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:20:49.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-02-05T05:20:49.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:20:49.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-02-05T05:20:50.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:20:50.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:20:50.893 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'ecpool' created 2024-02-05T05:20:50.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:20:51.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-02-05T05:20:51.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:20:51.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:20:51.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:20:51.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:20:51.915 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:20:51.916 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:20:51.916 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:20:51.916 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:20:51.916 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:20:51.916 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:20:52.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:20:52.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:20:52.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:20:52.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:20:52.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:20:52.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:20:52.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:20:52.479 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:20:52.479 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:20:52.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:20:52.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:20:52.480 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:20:52.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-02-05T05:20:52.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-02-05T05:20:52.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-02-05T05:20:52.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:20:52.621 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:20:52.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476757 2024-02-05T05:20:52.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476757 2024-02-05T05:20:52.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-68719476757' 2024-02-05T05:20:52.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:20:52.821 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:20:52.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855367 2024-02-05T05:20:52.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855367 2024-02-05T05:20:52.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-68719476757 2-150323855367' 2024-02-05T05:20:52.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:20:52.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-02-05T05:20:52.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:20:52.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:20:52.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-02-05T05:20:52.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:20:52.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-02-05T05:20:52.974 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836504 2024-02-05T05:20:52.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-02-05T05:20:52.974 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:20:53.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-02-05T05:20:53.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:20:54.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:20:54.275 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:20:54.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836504 2024-02-05T05:20:54.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:20:54.623 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476757 2024-02-05T05:20:54.623 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:20:54.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:20:54.626 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476757 2024-02-05T05:20:54.626 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:20:54.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476757 2024-02-05T05:20:54.628 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 68719476757 2024-02-05T05:20:54.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476757' 2024-02-05T05:20:54.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:20:55.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476757 -lt 68719476757 2024-02-05T05:20:55.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:20:55.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-150323855367 2024-02-05T05:20:55.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:20:55.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:20:55.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-150323855367 2024-02-05T05:20:55.007 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:20:55.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855367 2024-02-05T05:20:55.008 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 150323855367 2024-02-05T05:20:55.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 150323855367' 2024-02-05T05:20:55.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:20:55.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855367 -lt 150323855367 2024-02-05T05:20:55.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:20:55.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:20:55.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:20:55.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:20:55.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:20:55.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:20:55.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:20:55.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:20:55.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:20:55.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:20:55.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:20:56.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:20:56.079 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:20:56.080 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:20:56.080 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:20:56.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:20:56.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:20:56.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:20:56.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:20:56.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-02-05T05:20:56.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-02-05T05:20:56.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:20:56.940 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-02-05T05:20:56.940 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-02-05T05:20:56.940 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:20:56.941 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-02-05T05:20:56.941 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T05:20:56.941 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:20:56.942 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:20:56.942 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:20:57.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-02-05T05:20:57.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:20:57.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-02-05T05:20:57.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:20:57.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:20:57.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:20:57.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:20:57.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:20:57.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:20:57.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:20:57.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:20:57.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:20:57.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:20:57.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:20:57.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:20:57.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:20:57.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:20:57.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:20:57.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:20:57.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:20:57.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:20:57.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:20:57.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:20:57.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:20:57.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:20:57.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:20:57.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:20:59.876 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-02-05T05:20:59.876 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-02-05T05:20:59.876 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:21:00.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:21:00.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:21:00.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:21:00.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:21:00.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:21:00.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:21:00.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:21:00.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:21:00.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:21:00.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:21:00.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:21:00.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:21:00.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:21:00.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:21:00.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:21:00.431 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:21:00.431 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:21:00.431 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:21:00.432 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:21:00.433 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:21:00.433 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:21:00.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:21:00.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:21:00.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:21:00.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:21:00.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:21:00.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:21:00.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:21:00.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:21:00.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:21:00.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:21:00.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:21:00.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:21:00.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:21:00.434 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:21:00.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:21:00.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:21:00.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:21:00.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:21:00.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:21:00.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:21:00.475 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:00.471+0000 7f3002fc0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:00.503 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:00.503+0000 7f3002fc0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:00.504 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:00.503+0000 7f3002fc0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:00.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:21:00.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:21:00.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:21:00.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:21:00.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:21:00.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:00.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:21:00.755 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:21:00.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:00.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:21:01.038 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:01.035+0000 7f3002fc0640 -1 Falling back to public interface 2024-02-05T05:21:01.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:02.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:02.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:02.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:21:02.105 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:21:02.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:02.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:21:02.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:03.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:03.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:03.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:21:03.449 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:21:03.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:03.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:21:03.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:03.851 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:03.851+0000 7f3002fc0640 -1 osd.1 56 log_to_monitors true 2024-02-05T05:21:04.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:04.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:04.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:21:04.755 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:21:04.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:04.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:21:05.080 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 62 up_thru 62 down_at 57 last_clean_interval [16,56) [v2:127.0.0.1:6810/942903059,v1:127.0.0.1:6811/942903059] [v2:127.0.0.1:6812/942903059,v1:127.0.0.1:6813/942903059] exists,up 9430a457-82bb-43d8-85de-e1cd0b008c61 2024-02-05T05:21:05.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:21:05.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:21:05.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:21:05.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:21:05.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:21:05.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:21:05.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:21:05.081 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:21:05.081 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:21:05.081 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:21:05.081 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:21:05.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:21:05.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:21:05.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:21:05.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:21:05.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:21:05.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:21:05.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:21:05.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:21:05.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:21:05.612 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:21:05.612 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:21:05.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:21:05.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:21:05.612 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:21:05.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-02-05T05:21:05.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-02-05T05:21:05.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-02-05T05:21:05.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:21:05.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:21:06.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972354 2024-02-05T05:21:06.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972354 2024-02-05T05:21:06.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-266287972354' 2024-02-05T05:21:06.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:21:06.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:21:06.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855370 2024-02-05T05:21:06.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855370 2024-02-05T05:21:06.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-266287972354 2-150323855370' 2024-02-05T05:21:06.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:21:06.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-02-05T05:21:06.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:21:06.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:21:06.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-02-05T05:21:06.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:21:06.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-02-05T05:21:06.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-02-05T05:21:06.338 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836508 2024-02-05T05:21:06.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:21:06.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836508 2024-02-05T05:21:06.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:21:07.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:21:07.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:21:08.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-02-05T05:21:08.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:21:08.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-266287972354 2024-02-05T05:21:08.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:21:08.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:21:08.025 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-266287972354 2024-02-05T05:21:08.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:21:08.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972354 2024-02-05T05:21:08.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 266287972354' 2024-02-05T05:21:08.027 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 266287972354 2024-02-05T05:21:08.027 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:21:08.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972354 -lt 266287972354 2024-02-05T05:21:08.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:21:08.392 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-150323855370 2024-02-05T05:21:08.392 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:21:08.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:21:08.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-150323855370 2024-02-05T05:21:08.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:21:08.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855370 2024-02-05T05:21:08.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 150323855370' 2024-02-05T05:21:08.397 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 150323855370 2024-02-05T05:21:08.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:21:08.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855371 -lt 150323855370 2024-02-05T05:21:08.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:21:08.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:21:08.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:21:09.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:21:09.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:21:09.076 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:21:09.076 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:21:09.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:21:09.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:21:09.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:21:09.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:21:09.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:21:09.367 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:21:09.367 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:21:09.367 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:21:09.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:21:09.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T05:21:09.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:21:09.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:21:09.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:21:09.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:21:09.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:21:09.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:21:09.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:21:09.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:21:09.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:21:09.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:21:09.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:21:09.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:21:10.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:21:10.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:21:10.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:21:10.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:21:10.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:21:10.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:21:10.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:21:10.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-02-05T05:21:10.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:21:10.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:21:10.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:21:10.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:21:10.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-02-05T05:21:10.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-02-05T05:21:10.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:21:10.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:21:10.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:21:10.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:21:11.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-02-05T05:20:56.022581+0000 2024-02-05T05:21:11.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:21:11.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:20:56.022581+0000 2024-02-05T05:21:11.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:21:11.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:21:11.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:21:11.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:21:11.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:21:11.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:21:11.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:21:11.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:21:11.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:20:56.022581+0000 '>' 2024-02-05T05:20:56.022581+0000 2024-02-05T05:21:11.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:21:12.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:21:12.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:21:12.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:21:12.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:21:12.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:21:12.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:21:12.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:21:12.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:21:08.988701+0000 '>' 2024-02-05T05:20:56.022581+0000 2024-02-05T05:21:12.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:21:12.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-02-05T05:21:12.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:21:12.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:21:12.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:21:12.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:21:12.871 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:21:12.871 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:21:12.872 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:21:12.872 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:21:12.872 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:21:12.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:21:12.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:21:12.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:21:12.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:21:12.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:21:12.968 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:21:13.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:21:13.349 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:21:13.349 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:21:13.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:21:13.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:21:13.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:21:13.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836511 2024-02-05T05:21:13.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836511 2024-02-05T05:21:13.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511' 2024-02-05T05:21:13.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:21:13.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:21:13.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972356 2024-02-05T05:21:13.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972356 2024-02-05T05:21:13.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-266287972356' 2024-02-05T05:21:13.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:21:13.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:21:13.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855373 2024-02-05T05:21:13.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855373 2024-02-05T05:21:13.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-266287972356 2-150323855373' 2024-02-05T05:21:13.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:21:13.850 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:21:13.850 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836511 2024-02-05T05:21:13.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:21:13.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836511 2024-02-05T05:21:13.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:21:13.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836511 2024-02-05T05:21:13.853 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836511 2024-02-05T05:21:13.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-02-05T05:21:13.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:21:14.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836511 2024-02-05T05:21:14.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:21:14.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-266287972356 2024-02-05T05:21:14.241 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:21:14.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:21:14.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-266287972356 2024-02-05T05:21:14.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:21:14.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972356 2024-02-05T05:21:14.245 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 266287972356 2024-02-05T05:21:14.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 266287972356' 2024-02-05T05:21:14.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:21:14.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972356 -lt 266287972356 2024-02-05T05:21:14.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:21:14.600 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-150323855373 2024-02-05T05:21:14.600 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:21:14.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:21:14.602 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:21:14.602 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-150323855373 2024-02-05T05:21:14.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855373 2024-02-05T05:21:14.604 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 150323855373 2024-02-05T05:21:14.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 150323855373' 2024-02-05T05:21:14.604 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:21:14.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855372 -lt 150323855373 2024-02-05T05:21:14.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:21:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:21:15.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:21:16.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855373 -lt 150323855373 2024-02-05T05:21:16.308 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:21:16.309 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:21:16.309 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:21:16.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:21:16.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:21:16.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:21:16.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:21:16.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:21:16.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:21:16.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:21:16.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:21:17.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:21:17.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:21:17.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:21:17.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:21:17.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:21:17.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:21:17.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:21:17.404 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-02-05T05:21:17.404 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-02-05T05:21:17.404 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:21:17.404 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-02-05T05:21:17.404 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T05:21:17.404 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:21:17.404 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:21:17.405 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:21:17.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-02-05T05:21:17.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:21:17.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-02-05T05:21:18.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:21:18.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:21:18.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:21:18.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:21:18.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:21:18.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:21:18.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:21:18.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:21:18.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:21:18.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:21:18.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:21:18.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:21:18.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:21:18.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:21:18.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:21:18.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:21:19.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:21:19.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:21:19.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:21:19.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:21:19.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:21:19.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:21:19.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:21:19.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:21:20.787 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-02-05T05:21:20.787 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-02-05T05:21:20.787 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:21:20.787 INFO:tasks.workunit.client.0.smithi148.stdout:hinfo_key 2024-02-05T05:21:20.787 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:21:21.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:21:21.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:21:21.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:21:21.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:21:21.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:21:21.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:21:21.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:21:21.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:21:21.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:21:21.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:21:21.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:21:21.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:21:21.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:21:21.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:21:21.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:21:21.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:21:21.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:21:21.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:21:21.323 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:21:21.323 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:21:21.323 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:21:21.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:21:21.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:21:21.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:21:21.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:21:21.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:21:21.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:21:21.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:21:21.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:21:21.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:21:21.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:21:21.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:21:21.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:21:21.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:21:21.325 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:21:21.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:21:21.325 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:21:21.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:21:21.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:21:21.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:21:21.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:21:21.362 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:21.359+0000 7f54a9289640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:21.365 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:21.363+0000 7f54a9289640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:21.366 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:21.363+0000 7f54a9289640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:21:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:21:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:21:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:21:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:21:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:21:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:21:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:21:21.910 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:21.907+0000 7f54a9289640 -1 Falling back to public interface 2024-02-05T05:21:21.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:22.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:22.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:22.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:21:22.986 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:21:22.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:22.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:21:23.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:24.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:24.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:24.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:21:24.297 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:21:24.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:24.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:21:24.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:24.854 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:24.851+0000 7f54a9289640 -1 osd.1 71 log_to_monitors true 2024-02-05T05:21:25.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:25.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:25.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:21:25.615 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:21:25.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:25.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:21:26.014 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 78 up_thru 78 down_at 72 last_clean_interval [62,71) [v2:127.0.0.1:6810/1631825097,v1:127.0.0.1:6811/1631825097] [v2:127.0.0.1:6812/1631825097,v1:127.0.0.1:6813/1631825097] exists,up 9430a457-82bb-43d8-85de-e1cd0b008c61 2024-02-05T05:21:26.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:21:26.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:21:26.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:21:26.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:21:26.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:21:26.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:21:26.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:21:26.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:21:26.015 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:21:26.015 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:21:26.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:21:26.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:21:26.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:21:26.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:21:26.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:21:26.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:21:26.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:21:26.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:21:26.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:21:26.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:21:26.557 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:21:26.557 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:21:26.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:21:26.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:21:26.557 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:21:26.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836514 2024-02-05T05:21:26.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836514 2024-02-05T05:21:26.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514' 2024-02-05T05:21:26.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:21:26.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:21:26.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449090 2024-02-05T05:21:26.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449090 2024-02-05T05:21:26.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-335007449090' 2024-02-05T05:21:26.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:21:26.979 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:21:27.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855377 2024-02-05T05:21:27.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855377 2024-02-05T05:21:27.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-335007449090 2-150323855377' 2024-02-05T05:21:27.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:21:27.129 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836514 2024-02-05T05:21:27.129 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:21:27.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:21:27.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836514 2024-02-05T05:21:27.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:21:27.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836514 2024-02-05T05:21:27.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-02-05T05:21:27.132 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836514 2024-02-05T05:21:27.132 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:21:27.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836514 2024-02-05T05:21:27.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:21:28.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:21:28.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:21:28.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836514 2024-02-05T05:21:28.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:21:28.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-335007449090 2024-02-05T05:21:28.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:21:28.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:21:28.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-335007449090 2024-02-05T05:21:28.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:21:28.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449090 2024-02-05T05:21:28.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 335007449090' 2024-02-05T05:21:28.783 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 335007449090 2024-02-05T05:21:28.784 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:21:29.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449090 -lt 335007449090 2024-02-05T05:21:29.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:21:29.097 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-150323855377 2024-02-05T05:21:29.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:21:29.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:21:29.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-150323855377 2024-02-05T05:21:29.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:21:29.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855377 2024-02-05T05:21:29.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 150323855377' 2024-02-05T05:21:29.101 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 150323855377 2024-02-05T05:21:29.102 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:21:29.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855377 -lt 150323855377 2024-02-05T05:21:29.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:21:29.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:21:29.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:21:29.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:21:29.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:21:29.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:21:29.854 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:21:29.854 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:21:29.854 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:21:29.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:21:29.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:21:30.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:21:30.200 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:21:30.200 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:21:30.200 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:21:30.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:21:30.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:21:30.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:21:30.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:21:30.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:21:30.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:21:30.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:21:30.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:21:30.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:21:30.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:21:30.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:21:30.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:21:30.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:21:30.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:21:30.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:21:30.952 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:21:30.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:21:30.955 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:21:30.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:21:30.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:21:30.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:21:30.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:21:30.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:21:30.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:21:30.958 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:21:30.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:21:30.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:21:30.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:21:30.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:21:30.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:21:30.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:21:30.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:21:30.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:21:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:21:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:21:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:21:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:21:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:21:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:21:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:21:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:21:30.988 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:21:30.988 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:21:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:21:30.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:21:30.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:21:30.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:21:30.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:21:30.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:21:30.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:21:30.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:21:30.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:21:30.993 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:21:30.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:21:30.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:21:30.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:21:30.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:21:30.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:21:30.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:21:30.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:21:30.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:21:30.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:21:30.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:21:30.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:21:31.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:21:31.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:21:31.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:21:31.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:21:31.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:21:31.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:21:31.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T05:21:31.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:21:31.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:21:31.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:21:31.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:21:31.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-02-05T05:21:31.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-02-05T05:21:31.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-02-05T05:21:31.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T05:21:31.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-02-05T05:21:31.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-02-05T05:21:31.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-02-05T05:21:31.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:21:31.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:21:31.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:21:31.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:21:31.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:21:31.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:21:31.408 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:21:31.408 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:21:31.408 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:21:31.409 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:21:31.409 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:21:31.409 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:21:31.409 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:21:31.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:21:31.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:21:31.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:21:31.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:21:31.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:21:31.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:21:31.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:21:31.611 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:21:31.611 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:21:31.611 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:21:31.612 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:21:31.612 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:21:31.612 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:21:31.612 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:21:31.612 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:21:31.612 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T05:21:31.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:21:31.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:21:31.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:21:31.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:21:31.725 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:21:31.725 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:21:31.725 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:21:31.725 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:21:31.726 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:21:31.726 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:21:31.726 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:21:31.726 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:21:31.727 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:21:31.727 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T05:21:31.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-02-05T05:21:31.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:21:31.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:21:31.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:21:31.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:21:31.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:21:31.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:21:32.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:21:32.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:21:32.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:21:32.719 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:21:32.719 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:21:32.719 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:21:32.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:21:32.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:21:32.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:21:32.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:21:32.756 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-02-05T05:21:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:21:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:21:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:21:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:21:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:21:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:21:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:21:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:21:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:21:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:21:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:21:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:21:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:21:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:21:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:21:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:21:32.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:21:32.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:21:32.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:21:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:21:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:21:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:21:32.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:21:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:21:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:21:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:21:32.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:21:32.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:21:32.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:21:32.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:21:32.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:21:32.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:21:32.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:21:32.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:21:32.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:21:32.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:21:32.763 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:21:32.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ddcb8a2a-dc14-49b9-bb08-026e8a58279b 2024-02-05T05:21:32.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ddcb8a2a-dc14-49b9-bb08-026e8a58279b' 2024-02-05T05:21:32.763 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 ddcb8a2a-dc14-49b9-bb08-026e8a58279b 2024-02-05T05:21:32.763 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:21:32.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBccMBlz0GYLhAAMVhxKQS9FS5dLAspCZ91vw== 2024-02-05T05:21:32.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBccMBlz0GYLhAAMVhxKQS9FS5dLAspCZ91vw=="}' 2024-02-05T05:21:32.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ddcb8a2a-dc14-49b9-bb08-026e8a58279b -i td/osd-scrub-repair/0/new.json 2024-02-05T05:21:33.039 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:21:33.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:21:33.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBccMBlz0GYLhAAMVhxKQS9FS5dLAspCZ91vw== --osd-uuid ddcb8a2a-dc14-49b9-bb08-026e8a58279b 2024-02-05T05:21:33.094 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:33.090+0000 7f0e7a0c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:33.106 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:33.102+0000 7f0e7a0c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:33.107 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:33.106+0000 7f0e7a0c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:33.108 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:33.106+0000 7f0e7a0c0640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:21:33.108 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:33.106+0000 7f0e7a0c0640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:21:33.108 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:33.106+0000 7f0e7a0c0640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:21:33.206 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:33.202+0000 7f0e7a0c0640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:21:39.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:21:39.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:21:39.420 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:21:39.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:21:39.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:21:39.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:21:39.807 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:21:39.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:21:39.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:21:39.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:21:39.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:21:39.844 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:39.842+0000 7f72906d0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:39.883 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:39.882+0000 7f72906d0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:39.885 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:39.882+0000 7f72906d0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:40.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:21:40.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:21:40.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:21:40.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:21:40.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:21:40.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:40.090 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:21:40.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:21:40.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:21:40.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:40.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:40.470 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:40.466+0000 7f72906d0640 -1 Falling back to public interface 2024-02-05T05:21:41.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:41.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:41.402 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:21:41.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:21:41.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:41.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:21:41.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:42.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:42.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:42.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:21:42.702 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:21:42.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:42.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:21:43.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:43.081 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:21:43.078+0000 7f72906d0640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:21:44.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:44.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:44.014 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:21:44.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:21:44.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:44.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:21:44.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:45.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:45.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:45.453 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:21:45.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:21:45.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:45.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:21:45.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:46.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:46.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:46.758 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:21:46.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:21:46.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:46.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:21:47.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:48.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:48.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:48.145 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:21:48.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:21:48.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:48.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:21:48.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:49.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:49.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:49.463 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:21:49.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:21:49.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:49.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:21:49.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:50.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:50.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:50.857 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:21:50.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:21:50.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:50.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:21:51.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:52.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:52.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:52.231 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:21:52.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:21:52.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:52.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:21:52.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:53.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:53.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:53.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:21:53.554 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:21:53.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:53.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:21:53.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:54.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:54.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:54.863 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:21:54.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:21:54.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:54.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:21:55.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:56.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:56.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:56.259 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:21:56.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:21:56.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:56.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:21:56.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:57.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:57.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:57.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:21:57.576 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:21:57.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:57.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:21:58.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:59.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:59.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:59.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:21:59.308 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:21:59.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:59.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:22:00.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:01.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:01.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:01.367 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:22:01.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:22:01.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:01.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:22:02.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:03.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:03.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:03.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:22:03.246 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:22:03.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:03.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:22:04.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:05.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:05.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:05.116 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:22:05.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:22:05.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:05.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:22:06.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:07.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:07.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:07.017 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:22:07.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:22:07.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:07.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:22:07.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:08.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:08.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:08.939 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:22:08.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:22:08.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:08.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:22:09.287 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2716914018,v1:127.0.0.1:6803/2716914018] [v2:127.0.0.1:6804/2716914018,v1:127.0.0.1:6805/2716914018] exists,up ddcb8a2a-dc14-49b9-bb08-026e8a58279b 2024-02-05T05:22:09.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:22:09.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:22:09.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:22:09.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:22:09.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:22:09.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:22:09.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:22:09.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:22:09.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:22:09.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:22:09.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:22:09.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:22:09.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:22:09.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:22:09.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:22:09.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:22:09.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:22:09.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:22:09.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:22:09.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:22:09.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:22:09.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:22:09.294 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:22:09.294 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:22:09.294 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:22:09.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:22:09.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:22:09.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:22:09.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:22:09.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:22:09.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:22:09.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:22:09.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:22:09.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:22:09.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:22:09.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:22:09.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:22:09.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:22:09.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:22:09.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:22:09.295 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 b4332dd3-6121-47bd-a3df-b2f067286166 2024-02-05T05:22:09.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b4332dd3-6121-47bd-a3df-b2f067286166 2024-02-05T05:22:09.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b4332dd3-6121-47bd-a3df-b2f067286166' 2024-02-05T05:22:09.296 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:22:09.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCBcMBlCMTcEhAAG7vjlJ/b0PrTpOeVnb8pxQ== 2024-02-05T05:22:09.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCBcMBlCMTcEhAAG7vjlJ/b0PrTpOeVnb8pxQ=="}' 2024-02-05T05:22:09.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b4332dd3-6121-47bd-a3df-b2f067286166 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:22:09.761 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:22:09.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:22:09.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCBcMBlCMTcEhAAG7vjlJ/b0PrTpOeVnb8pxQ== --osd-uuid b4332dd3-6121-47bd-a3df-b2f067286166 2024-02-05T05:22:09.823 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:09.821+0000 7f24ef8c7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:09.871 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:09.869+0000 7f24ef8c7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:09.872 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:09.873+0000 7f24ef8c7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:09.873 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:09.873+0000 7f24ef8c7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:22:09.873 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:09.873+0000 7f24ef8c7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:22:09.873 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:09.873+0000 7f24ef8c7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:22:09.951 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:09.949+0000 7f24ef8c7640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:22:16.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:22:16.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:22:16.918 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:22:16.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:22:16.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:22:17.312 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:22:17.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:22:17.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:22:17.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:22:17.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:22:17.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:22:17.345 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:17.345+0000 7f8285a01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:17.378 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:17.377+0000 7f8285a01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:17.380 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:17.381+0000 7f8285a01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:17.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:22:17.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:22:17.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:22:17.652 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:22:17.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:22:17.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:22:17.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:17.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:22:17.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:17.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:17.934 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:17.933+0000 7f8285a01640 -1 Falling back to public interface 2024-02-05T05:22:18.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:19.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:19.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:19.024 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:22:19.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:22:19.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:19.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:19.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:20.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:20.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:20.335 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:22:20.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:22:20.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:20.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:20.603 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:20.601+0000 7f8285a01640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:22:20.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:21.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:21.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:21.650 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:22:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:22:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:21.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:22.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:22.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:22.990 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:22:22.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:22:22.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:22.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:23.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:24.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:24.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:24.563 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:22:24.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:22:24.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:24.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:24.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:25.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:25.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:25.883 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:22:25.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:22:25.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:25.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:26.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:27.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:27.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:27.279 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:22:27.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:22:27.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:27.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:27.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:28.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:28.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:28.644 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:22:28.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:22:28.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:28.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:28.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:29.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:29.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:29.960 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:22:29.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:22:29.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:29.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:30.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:31.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:31.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:31.423 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:22:31.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:22:31.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:31.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:31.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:32.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:32.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:32.917 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:22:32.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:22:32.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:32.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:33.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:34.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:34.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:34.317 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:22:34.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:22:34.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:34.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:34.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:35.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:35.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:35.738 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:22:35.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:22:35.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:35.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:36.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:37.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:37.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:37.136 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:22:37.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:22:37.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:37.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:37.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:38.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:38.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:38.669 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:22:38.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:22:38.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:38.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:39.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:40.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:40.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:40.181 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:22:40.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:22:40.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:40.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:41.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:42.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:42.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:42.441 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:22:42.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:22:42.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:42.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:43.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:44.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:44.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:44.123 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:22:44.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:22:44.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:44.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:44.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:45.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:45.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:45.451 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:22:45.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:22:45.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:45.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:46.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:47.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:47.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:47.148 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:22:47.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:22:47.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:47.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:22:47.463 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2058284179,v1:127.0.0.1:6811/2058284179] [v2:127.0.0.1:6812/2058284179,v1:127.0.0.1:6813/2058284179] exists,up b4332dd3-6121-47bd-a3df-b2f067286166 2024-02-05T05:22:47.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:22:47.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:22:47.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:22:47.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:22:47.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:22:47.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:22:47.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:22:47.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:22:47.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:22:47.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:22:47.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:22:47.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:22:47.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:22:47.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:22:47.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:22:47.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:22:47.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:22:47.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:22:47.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:22:47.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:22:47.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:22:47.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:22:47.467 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:22:47.467 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:22:47.467 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:22:47.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:22:47.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:22:47.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:22:47.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:22:47.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:22:47.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:22:47.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:22:47.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:22:47.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:22:47.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:22:47.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:22:47.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:22:47.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:22:47.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:22:47.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:22:47.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=632f6f74-622f-4fba-ac27-e49b1133a0be 2024-02-05T05:22:47.470 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 632f6f74-622f-4fba-ac27-e49b1133a0be 2024-02-05T05:22:47.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 632f6f74-622f-4fba-ac27-e49b1133a0be' 2024-02-05T05:22:47.471 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:22:47.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCncMBlwsJfHRAA2T54G1m0PMKmF2yNEQdi2g== 2024-02-05T05:22:47.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCncMBlwsJfHRAA2T54G1m0PMKmF2yNEQdi2g=="}' 2024-02-05T05:22:47.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 632f6f74-622f-4fba-ac27-e49b1133a0be -i td/osd-scrub-repair/2/new.json 2024-02-05T05:22:47.850 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:22:47.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:22:47.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCncMBlwsJfHRAA2T54G1m0PMKmF2yNEQdi2g== --osd-uuid 632f6f74-622f-4fba-ac27-e49b1133a0be 2024-02-05T05:22:47.911 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:47.908+0000 7f4f01b22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:47.952 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:47.952+0000 7f4f01b22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:47.953 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:47.952+0000 7f4f01b22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:47.954 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:47.952+0000 7f4f01b22640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:22:47.954 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:47.952+0000 7f4f01b22640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:22:47.954 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:47.952+0000 7f4f01b22640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:22:48.025 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:48.024+0000 7f4f01b22640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:22:55.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:22:55.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:22:55.487 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T05:22:55.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:22:55.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:22:55.872 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T05:22:55.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:22:55.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:22:55.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:22:55.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:22:55.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:22:55.909 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:55.908+0000 7f6c21e5f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:55.998 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:55.996+0000 7f6c21e5f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:56.000 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:56.000+0000 7f6c21e5f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:56.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:22:56.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:22:56.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:22:56.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:22:56.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:22:56.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:56.216 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:22:56.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:22:56.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:56.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:22:56.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:56.578 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:56.576+0000 7f6c21e5f640 -1 Falling back to public interface 2024-02-05T05:22:57.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:57.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:57.523 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:22:57.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:22:57.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:57.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:22:57.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:58.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:58.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:58.824 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:22:58.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:22:58.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:58.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:22:59.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:59.642 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:22:59.640+0000 7f6c21e5f640 -1 osd.2 0 log_to_monitors true 2024-02-05T05:23:00.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:00.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:00.331 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:23:00.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:23:00.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:00.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:00.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:01.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:01.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:01.759 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:23:01.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:23:01.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:01.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:02.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:03.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:03.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:03.244 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:23:03.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:23:03.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:03.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:03.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:04.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:04.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:04.759 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:23:04.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:23:04.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:04.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:05.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:06.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:06.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:06.182 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:23:06.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:23:06.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:06.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:06.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:07.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:07.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:07.580 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:23:07.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:23:07.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:07.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:08.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:09.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:09.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:09.033 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:23:09.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:23:09.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:09.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:09.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:10.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:10.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:10.354 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:23:10.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:23:10.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:10.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:10.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:11.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:11.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:11.812 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:23:11.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:23:11.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:11.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:12.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:13.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:13.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:13.193 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:23:13.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:23:13.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:13.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:13.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:14.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:14.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:14.604 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:23:14.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:23:14.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:14.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:14.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:15.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:15.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:15.928 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:23:15.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:23:15.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:15.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:16.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:17.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:17.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:17.363 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:23:17.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:23:17.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:17.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:18.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:19.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:19.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:19.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:23:19.335 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:23:19.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:19.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:20.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:21.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:21.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:21.084 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:23:21.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:23:21.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:21.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:21.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:22.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:22.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:22.807 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:23:22.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:23:22.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:22.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:23.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:24.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:24.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:24.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:23:24.712 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:23:24.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:24.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:25.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:26.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:26.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:26.467 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:23:26.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:23:26.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:26.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:26.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:27.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:27.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:27.874 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T05:23:27.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T05:23:27.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:27.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:23:28.193 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1307766647,v1:127.0.0.1:6819/1307766647] [v2:127.0.0.1:6820/1307766647,v1:127.0.0.1:6821/1307766647] exists,up 632f6f74-622f-4fba-ac27-e49b1133a0be 2024-02-05T05:23:28.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:23:28.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:23:28.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:23:28.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-02-05T05:23:28.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:23:28.515 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T05:23:28.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:23:28.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:23:28.946 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T05:23:28.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:23:29.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:23:32.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-02-05T05:23:32.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:23:32.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:23:32.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:23:32.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:23:32.047 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:23:32.047 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:23:32.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:23:32.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:23:32.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:23:32.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:23:32.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:23:32.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:23:32.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:23:32.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:23:32.174 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:23:32.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:23:32.487 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:23:32.488 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:23:32.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:23:32.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:23:32.488 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:23:32.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-02-05T05:23:32.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-02-05T05:23:32.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-02-05T05:23:32.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:23:32.764 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:23:32.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=73014444044 2024-02-05T05:23:32.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 73014444044 2024-02-05T05:23:32.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-73014444044' 2024-02-05T05:23:32.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:23:32.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:23:33.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822670 2024-02-05T05:23:33.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822670 2024-02-05T05:23:33.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-73014444044 2-154618822670' 2024-02-05T05:23:33.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:23:33.054 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-02-05T05:23:33.054 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:23:33.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:23:33.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-02-05T05:23:33.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:23:33.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-02-05T05:23:33.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-02-05T05:23:33.058 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836500 2024-02-05T05:23:33.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:23:33.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-02-05T05:23:33.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:23:33.371 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-73014444044 2024-02-05T05:23:33.371 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:23:33.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:23:33.373 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-73014444044 2024-02-05T05:23:33.374 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:23:33.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=73014444044 2024-02-05T05:23:33.375 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 73014444044 2024-02-05T05:23:33.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 73014444044' 2024-02-05T05:23:33.375 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:23:33.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 73014444044 -lt 73014444044 2024-02-05T05:23:33.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:23:33.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:23:33.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-154618822670 2024-02-05T05:23:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:23:33.716 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-154618822670 2024-02-05T05:23:33.716 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:23:33.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822670 2024-02-05T05:23:33.717 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 154618822670 2024-02-05T05:23:33.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 154618822670' 2024-02-05T05:23:33.717 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:23:34.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822670 -lt 154618822670 2024-02-05T05:23:34.121 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:23:34.122 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:23:34.122 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:23:34.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:23:34.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:23:34.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:23:34.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:23:34.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:23:34.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:23:34.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:23:34.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:23:34.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:23:34.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:23:34.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:23:34.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:23:35.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:23:35.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:23:35.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:23:35.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-02-05T05:23:35.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:23:35.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:23:35.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-02-05T05:23:35.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:23:35.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-02-05T05:23:35.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:23:35.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:23:36.715 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'ecpool' created 2024-02-05T05:23:36.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:23:37.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-02-05T05:23:37.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-02-05T05:23:38.533 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 2 allow_ec_overwrites to true 2024-02-05T05:23:38.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:23:38.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:23:38.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:23:38.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:23:38.556 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:23:38.557 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:23:38.557 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:23:38.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:23:38.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:23:38.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:23:38.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:23:38.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:23:38.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:23:38.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:23:38.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:23:38.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:23:39.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:23:39.072 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:23:39.072 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:23:39.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:23:39.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:23:39.072 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:23:39.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-02-05T05:23:39.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-02-05T05:23:39.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-02-05T05:23:39.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:23:39.309 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:23:39.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=73014444047 2024-02-05T05:23:39.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 73014444047 2024-02-05T05:23:39.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-73014444047' 2024-02-05T05:23:39.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:23:39.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:23:39.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822673 2024-02-05T05:23:39.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822673 2024-02-05T05:23:39.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-73014444047 2-154618822673' 2024-02-05T05:23:39.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:23:39.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-02-05T05:23:39.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:23:39.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:23:39.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-02-05T05:23:39.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:23:39.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-02-05T05:23:39.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-02-05T05:23:39.596 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836503 2024-02-05T05:23:39.597 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:23:39.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-02-05T05:23:39.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:23:40.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:23:40.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:23:41.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-02-05T05:23:41.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:23:42.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:23:42.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:23:42.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-02-05T05:23:42.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:23:42.508 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-73014444047 2024-02-05T05:23:42.508 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:23:42.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:23:42.510 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-73014444047 2024-02-05T05:23:42.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:23:42.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=73014444047 2024-02-05T05:23:42.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 73014444047' 2024-02-05T05:23:42.512 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 73014444047 2024-02-05T05:23:42.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:23:42.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 73014444047 -lt 73014444047 2024-02-05T05:23:42.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:23:42.928 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-154618822673 2024-02-05T05:23:42.928 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:23:42.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:23:42.928 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-154618822673 2024-02-05T05:23:42.929 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:23:42.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822673 2024-02-05T05:23:42.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 154618822673' 2024-02-05T05:23:42.930 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 154618822673 2024-02-05T05:23:42.931 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:23:43.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822673 -lt 154618822673 2024-02-05T05:23:43.256 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:23:43.256 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:23:43.257 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:23:43.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:23:43.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:23:43.631 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:23:43.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:23:43.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:23:43.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:23:43.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:23:43.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:23:43.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:23:43.939 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:23:43.939 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:23:43.940 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:23:44.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:23:44.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:23:44.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:23:44.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:23:44.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-02-05T05:23:44.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-02-05T05:23:44.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:23:44.667 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-02-05T05:23:44.668 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-02-05T05:23:44.668 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:23:44.669 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-02-05T05:23:44.669 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T05:23:44.669 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:23:44.669 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:23:44.669 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:23:44.968 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-02-05T05:23:44.969 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:23:44.969 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-02-05T05:23:45.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:23:45.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:23:45.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:23:45.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:23:45.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:23:45.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:23:45.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:23:45.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:23:45.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:23:45.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:23:45.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:23:45.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:23:45.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:23:45.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:23:45.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:23:45.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:23:45.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:23:45.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:23:45.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:23:45.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:23:45.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:23:45.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:23:45.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:23:45.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:23:47.561 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-02-05T05:23:47.562 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-02-05T05:23:47.562 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:23:48.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:23:48.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:23:48.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:23:48.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:23:48.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:23:48.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:23:48.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:23:48.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:23:48.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:23:48.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:23:48.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:23:48.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:23:48.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:23:48.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:23:48.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:23:48.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:23:48.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:23:48.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:23:48.101 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:23:48.102 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:23:48.102 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:23:48.102 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:23:48.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:23:48.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:23:48.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:23:48.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:23:48.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:23:48.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:23:48.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:23:48.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:23:48.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:23:48.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:23:48.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:23:48.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:23:48.103 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:23:48.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:23:48.103 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:23:48.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:23:48.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:23:48.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:23:48.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:23:48.140 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:23:48.139+0000 7f795c39b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:48.143 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:23:48.139+0000 7f795c39b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:48.144 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:23:48.143+0000 7f795c39b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:48.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:23:48.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:23:48.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:23:48.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:23:48.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:23:48.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:48.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:23:48.423 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:23:48.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:48.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:23:48.690 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:23:48.687+0000 7f795c39b640 -1 Falling back to public interface 2024-02-05T05:23:48.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:49.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:49.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:49.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:23:49.753 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:23:49.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:49.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:23:50.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:51.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:51.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:51.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:23:51.101 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:23:51.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:51.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:23:51.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:51.521 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:23:51.519+0000 7f795c39b640 -1 osd.1 58 log_to_monitors true 2024-02-05T05:23:52.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:52.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:52.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:23:52.439 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:23:52.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:52.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:23:52.827 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 65 up_thru 65 down_at 59 last_clean_interval [17,58) [v2:127.0.0.1:6810/2553363985,v1:127.0.0.1:6811/2553363985] [v2:127.0.0.1:6812/2553363985,v1:127.0.0.1:6813/2553363985] exists,up b4332dd3-6121-47bd-a3df-b2f067286166 2024-02-05T05:23:52.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:23:52.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:23:52.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:23:52.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:23:52.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:23:52.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:23:52.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:23:52.828 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:23:52.829 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:23:52.830 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:23:52.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:23:52.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:23:52.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:23:52.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:23:52.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:23:52.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:23:52.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:23:52.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:23:52.938 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:23:53.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:23:53.358 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:23:53.358 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:23:53.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:23:53.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:23:53.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:23:53.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-02-05T05:23:53.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-02-05T05:23:53.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-02-05T05:23:53.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:23:53.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:23:53.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874242 2024-02-05T05:23:53.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874242 2024-02-05T05:23:53.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-279172874242' 2024-02-05T05:23:53.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:23:53.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:23:54.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822677 2024-02-05T05:23:54.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822677 2024-02-05T05:23:54.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-279172874242 2-154618822677' 2024-02-05T05:23:54.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:23:54.049 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-02-05T05:23:54.049 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:23:54.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:23:54.051 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:23:54.051 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-02-05T05:23:54.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-02-05T05:23:54.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-02-05T05:23:54.053 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836506 2024-02-05T05:23:54.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:23:54.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836506 2024-02-05T05:23:54.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:23:55.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:23:55.418 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:23:55.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836506 2024-02-05T05:23:55.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:23:55.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874242 2024-02-05T05:23:55.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:23:55.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:23:55.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874242 2024-02-05T05:23:55.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:23:55.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874242 2024-02-05T05:23:55.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874242' 2024-02-05T05:23:55.750 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 279172874242 2024-02-05T05:23:55.751 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:23:56.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874242 -lt 279172874242 2024-02-05T05:23:56.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:23:56.122 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-154618822677 2024-02-05T05:23:56.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:23:56.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:23:56.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-154618822677 2024-02-05T05:23:56.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:23:56.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822677 2024-02-05T05:23:56.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 154618822677' 2024-02-05T05:23:56.127 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 154618822677 2024-02-05T05:23:56.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:23:56.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822677 -lt 154618822677 2024-02-05T05:23:56.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:23:56.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:23:56.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:23:56.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:23:56.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:23:56.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:23:56.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:23:56.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:23:56.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:23:56.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:23:56.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:23:57.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:23:57.241 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:23:57.241 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:23:57.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:23:57.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:23:57.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:23:57.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:23:57.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-02-05T05:23:57.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:23:57.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:23:57.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:23:57.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:23:58.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-02-05T05:23:58.073 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-02-05T05:23:58.074 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:23:58.074 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:23:58.074 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:23:58.074 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:23:58.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-02-05T05:23:41.577065+0000 2024-02-05T05:23:58.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:23:58.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:23:41.577065+0000 2024-02-05T05:23:58.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:23:58.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:23:58.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:23:58.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:23:58.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:23:58.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:23:58.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:23:58.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:23:58.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:23:41.577065+0000 '>' 2024-02-05T05:23:41.577065+0000 2024-02-05T05:23:58.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:23:59.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:23:59.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:23:59.681 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:23:59.681 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:23:59.682 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:23:59.682 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:23:59.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:24:00.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:23:55.965266+0000 '>' 2024-02-05T05:23:41.577065+0000 2024-02-05T05:24:00.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:24:00.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-02-05T05:24:00.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:24:00.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:24:00.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:24:00.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:24:00.015 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:24:00.016 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:24:00.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:24:00.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:24:00.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:24:00.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:24:00.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:24:00.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:24:00.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:24:00.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:24:00.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:24:00.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:24:00.545 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:24:00.545 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:24:00.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:24:00.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:24:00.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:24:00.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-02-05T05:24:00.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-02-05T05:24:00.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-02-05T05:24:00.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:24:00.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:24:00.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874244 2024-02-05T05:24:00.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874244 2024-02-05T05:24:00.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-279172874244' 2024-02-05T05:24:00.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:24:00.901 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:24:01.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822679 2024-02-05T05:24:01.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822679 2024-02-05T05:24:01.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-279172874244 2-154618822679' 2024-02-05T05:24:01.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:24:01.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-02-05T05:24:01.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:24:01.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:24:01.055 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-02-05T05:24:01.055 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:24:01.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-02-05T05:24:01.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-02-05T05:24:01.056 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836509 2024-02-05T05:24:01.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:24:01.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-02-05T05:24:01.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:24:01.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874244 2024-02-05T05:24:01.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:24:01.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:24:01.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874244 2024-02-05T05:24:01.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:24:01.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874244 2024-02-05T05:24:01.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874244' 2024-02-05T05:24:01.353 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 279172874244 2024-02-05T05:24:01.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:24:01.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874244 -lt 279172874244 2024-02-05T05:24:01.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:24:01.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-154618822679 2024-02-05T05:24:01.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:24:01.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:24:01.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:24:01.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-154618822679 2024-02-05T05:24:01.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822679 2024-02-05T05:24:01.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 154618822679' 2024-02-05T05:24:01.687 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 154618822679 2024-02-05T05:24:01.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:24:02.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822679 -lt 154618822679 2024-02-05T05:24:02.099 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:24:02.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:24:02.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:24:02.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:24:02.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:24:02.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:24:02.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:24:02.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:24:02.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:24:02.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:24:02.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:24:02.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:24:02.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:24:02.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:24:02.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:24:03.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:24:03.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:24:03.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:24:03.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-02-05T05:24:03.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-02-05T05:24:03.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:24:03.157 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-02-05T05:24:03.157 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T05:24:03.157 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:24:03.157 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:24:03.157 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:24:03.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-02-05T05:24:03.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:24:03.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-02-05T05:24:04.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:24:04.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:24:04.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:24:04.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:24:04.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:24:04.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:24:04.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:24:04.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:24:04.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:24:04.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:24:04.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:24:04.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:24:04.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:24:04.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:24:04.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:24:04.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:24:04.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:24:04.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:24:04.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:24:04.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:24:04.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:24:04.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:24:04.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:24:04.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:24:06.700 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-02-05T05:24:06.701 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-02-05T05:24:06.701 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:24:06.701 INFO:tasks.workunit.client.0.smithi148.stdout:hinfo_key 2024-02-05T05:24:06.701 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:24:07.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:24:07.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:24:07.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:24:07.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:24:07.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:24:07.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:24:07.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:24:07.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:24:07.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:24:07.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:24:07.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:24:07.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:24:07.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:24:07.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:24:07.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:24:07.003 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:24:07.003 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:24:07.003 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:24:07.004 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:24:07.004 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:24:07.004 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:24:07.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:24:07.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:24:07.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:24:07.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:24:07.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:24:07.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:24:07.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:24:07.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:24:07.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:24:07.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:24:07.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:24:07.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:24:07.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:24:07.005 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:24:07.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:24:07.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:24:07.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:24:07.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:24:07.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:24:07.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:24:07.044 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:07.042+0000 7fcf1ad4a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:07.046 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:07.042+0000 7fcf1ad4a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:07.047 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:07.046+0000 7fcf1ad4a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:07.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:24:07.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:24:07.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:24:07.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:24:07.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:24:07.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:07.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:24:07.322 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:24:07.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:07.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:24:07.582 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:07.578+0000 7fcf1ad4a640 -1 Falling back to public interface 2024-02-05T05:24:07.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:08.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:08.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:08.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:24:08.629 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:24:08.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:08.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:24:08.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:09.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:09.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:09.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:24:09.953 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:24:09.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:09.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:24:10.209 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:10.206+0000 7fcf1ad4a640 -1 osd.1 73 log_to_monitors true 2024-02-05T05:24:10.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:11.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:11.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:11.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:24:11.275 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:24:11.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:11.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:24:11.769 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 80 up_thru 80 down_at 74 last_clean_interval [65,73) [v2:127.0.0.1:6810/2630087432,v1:127.0.0.1:6811/2630087432] [v2:127.0.0.1:6812/2630087432,v1:127.0.0.1:6813/2630087432] exists,up b4332dd3-6121-47bd-a3df-b2f067286166 2024-02-05T05:24:11.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:24:11.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:24:11.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:24:11.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:24:11.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:24:11.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:24:11.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:24:11.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:24:11.771 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:24:11.771 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:24:11.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:24:11.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:24:11.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:24:11.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:24:11.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:24:11.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:24:11.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:24:11.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:24:11.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:24:12.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:24:12.359 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:24:12.359 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:24:12.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:24:12.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:24:12.359 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:24:12.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-02-05T05:24:12.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-02-05T05:24:12.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-02-05T05:24:12.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:24:12.510 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:24:12.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-02-05T05:24:12.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-02-05T05:24:12.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-343597383682' 2024-02-05T05:24:12.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:24:12.682 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:24:12.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822682 2024-02-05T05:24:12.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822682 2024-02-05T05:24:12.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-343597383682 2-154618822682' 2024-02-05T05:24:12.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:24:12.821 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-02-05T05:24:12.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:24:12.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:24:12.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-02-05T05:24:12.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:24:12.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-02-05T05:24:12.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-02-05T05:24:12.824 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836512 2024-02-05T05:24:12.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:24:13.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836512 2024-02-05T05:24:13.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:24:14.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:24:14.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:24:14.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-02-05T05:24:14.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:24:14.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-343597383682 2024-02-05T05:24:14.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:24:14.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:24:14.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-343597383682 2024-02-05T05:24:14.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:24:14.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-02-05T05:24:14.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2024-02-05T05:24:14.445 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 343597383682 2024-02-05T05:24:14.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:24:14.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383682 -lt 343597383682 2024-02-05T05:24:14.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:24:14.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-154618822682 2024-02-05T05:24:14.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:24:14.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:24:14.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-154618822682 2024-02-05T05:24:14.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:24:14.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822682 2024-02-05T05:24:14.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 154618822682' 2024-02-05T05:24:14.796 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 154618822682 2024-02-05T05:24:14.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:24:15.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822682 -lt 154618822682 2024-02-05T05:24:15.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:24:15.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:24:15.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:24:15.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:24:15.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:24:15.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:24:15.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:24:15.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:24:15.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:24:15.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:24:15.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:24:15.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:24:15.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:24:15.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:24:15.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:24:16.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:24:16.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:24:16.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:24:16.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:24:16.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:24:16.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:24:16.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:24:16.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:24:16.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:24:16.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:24:16.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:24:16.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:24:16.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:24:16.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:24:16.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:24:16.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:24:16.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:24:16.435 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:24:16.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:24:16.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:24:16.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:24:16.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:24:16.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:24:16.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:24:16.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:24:16.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:24:16.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:24:16.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:24:16.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:24:16.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:24:16.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:24:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:24:16.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:24:16.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:24:16.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:24:16.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:24:16.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:24:16.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:24:16.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:24:16.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:24:16.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:24:16.462 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:24:16.462 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:24:16.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:24:16.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:24:16.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:24:16.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:24:16.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:24:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:24:16.464 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:24:16.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:24:16.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:24:16.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:24:16.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:24:16.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:24:16.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:24:16.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:24:16.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:24:16.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:24:16.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:24:16.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:24:16.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:24:16.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:24:16.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:24:16.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:24:16.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:24:16.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:24:16.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:24:16.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:24:16.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:24:16.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T05:24:16.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:24:16.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:24:16.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:24:16.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:24:16.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-02-05T05:24:16.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-02-05T05:24:16.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-02-05T05:24:16.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-02-05T05:24:16.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-02-05T05:24:16.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-02-05T05:24:16.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:24:16.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:24:16.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:24:16.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:24:16.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:24:16.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:24:16.865 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:24:16.865 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:24:16.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:24:16.866 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:24:16.866 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:24:16.866 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:24:16.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:24:16.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:24:17.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:24:17.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:24:17.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:24:17.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:24:17.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:24:17.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:24:17.105 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:24:17.105 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:24:17.105 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:24:17.106 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:24:17.106 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:24:17.106 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:24:17.106 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:24:17.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:24:17.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T05:24:17.221 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:24:17.221 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:24:17.221 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:24:17.221 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:24:17.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:24:17.222 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:24:17.222 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:24:17.222 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:24:17.223 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:24:17.223 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:24:17.223 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:24:17.223 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:24:17.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:24:17.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T05:24:17.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-02-05T05:24:17.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:24:17.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:24:17.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:24:17.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:24:17.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:24:17.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:24:18.212 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:24:18.212 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:24:18.212 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:24:18.213 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:24:18.213 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:24:18.213 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:24:18.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:24:18.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:24:18.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:24:18.250 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-02-05T05:24:18.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:24:18.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-02-05T05:24:18.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:24:18.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:24:18.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:24:18.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:24:18.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:24:18.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:24:18.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:24:18.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:24:18.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:24:18.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:24:18.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:24:18.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:24:18.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:24:18.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:24:18.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:24:18.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:24:18.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:24:18.254 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:24:18.254 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:24:18.254 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:24:18.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:24:18.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:24:18.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:24:18.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:24:18.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:24:18.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:24:18.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:24:18.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:24:18.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:24:18.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:24:18.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:24:18.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:24:18.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:24:18.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:24:18.256 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:24:18.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b8b1e649-0b77-469d-85b9-a3003db11de1 2024-02-05T05:24:18.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b8b1e649-0b77-469d-85b9-a3003db11de1' 2024-02-05T05:24:18.257 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 b8b1e649-0b77-469d-85b9-a3003db11de1 2024-02-05T05:24:18.257 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:24:18.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACccBlWRxnEBAAo0uNEw7D/hEHD1pHOy7CkQ== 2024-02-05T05:24:18.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACccBlWRxnEBAAo0uNEw7D/hEHD1pHOy7CkQ=="}' 2024-02-05T05:24:18.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b8b1e649-0b77-469d-85b9-a3003db11de1 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:24:18.524 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:24:18.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:24:18.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACccBlWRxnEBAAo0uNEw7D/hEHD1pHOy7CkQ== --osd-uuid b8b1e649-0b77-469d-85b9-a3003db11de1 2024-02-05T05:24:18.571 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:18.566+0000 7ff8a1bad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:18.591 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:18.590+0000 7ff8a1bad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:18.593 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:18.590+0000 7ff8a1bad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:18.593 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:18.590+0000 7ff8a1bad640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:24:18.593 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:18.590+0000 7ff8a1bad640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:24:18.593 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:18.590+0000 7ff8a1bad640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:24:18.684 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:18.682+0000 7ff8a1bad640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:24:24.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:24:24.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:24:24.961 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:24:24.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:24:24.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:24:25.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:24:25.353 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:24:25.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:24:25.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:24:25.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:24:25.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:24:25.385 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:25.382+0000 7f88f3dc6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:25.417 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:25.414+0000 7f88f3dc6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:25.418 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:25.414+0000 7f88f3dc6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:25.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:24:25.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:24:25.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:24:25.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:24:25.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:24:25.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:25.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:24:25.680 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:24:25.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:25.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:25.974 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:25.970+0000 7f88f3dc6640 -1 Falling back to public interface 2024-02-05T05:24:26.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:27.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:27.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:27.026 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:24:27.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:24:27.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:27.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:27.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:28.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:28.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:28.355 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:24:28.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:24:28.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:28.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:28.623 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:28.622+0000 7f88f3dc6640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:24:28.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:29.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:29.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:29.673 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:24:29.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:24:29.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:29.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:29.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:30.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:30.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:30.991 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:24:30.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:24:30.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:30.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:31.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:32.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:32.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:32.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:24:32.311 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:24:32.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:32.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:32.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:33.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:33.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:33.792 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:24:33.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:24:33.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:33.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:34.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:35.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:35.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:35.105 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:24:35.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:24:35.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:35.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:35.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:36.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:36.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:36.417 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:24:36.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:24:36.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:36.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:36.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:37.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:37.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:37.795 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:24:37.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:24:37.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:37.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:38.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:39.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:39.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:39.123 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:24:39.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:24:39.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:39.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:39.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:40.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:40.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:40.435 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:24:40.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:24:40.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:40.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:40.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:41.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:41.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:41.758 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:24:41.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:24:41.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:41.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:42.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:43.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:43.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:43.078 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:24:43.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:24:43.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:43.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:43.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:44.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:44.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:44.517 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:24:44.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:24:44.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:44.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:45.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:46.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:46.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:46.072 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:24:46.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:24:46.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:46.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:46.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:47.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:47.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:47.531 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:24:47.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:24:47.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:47.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:48.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:49.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:49.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:49.359 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:24:49.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:24:49.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:49.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:50.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:51.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:51.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:51.220 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:24:51.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:24:51.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:51.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:52.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:53.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:53.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:53.046 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:24:53.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:24:53.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:53.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:53.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:54.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:54.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:54.357 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:24:54.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:24:54.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:54.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:24:54.678 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2404415568,v1:127.0.0.1:6803/2404415568] [v2:127.0.0.1:6804/2404415568,v1:127.0.0.1:6805/2404415568] exists,up b8b1e649-0b77-469d-85b9-a3003db11de1 2024-02-05T05:24:54.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:24:54.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:24:54.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:24:54.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:24:54.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-02-05T05:24:54.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:24:54.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:24:54.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:24:54.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:24:54.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:24:54.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:24:54.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:24:54.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:24:54.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:24:54.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:24:54.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:24:54.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:24:54.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:24:54.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:24:54.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:24:54.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:24:54.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:24:54.683 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:24:54.683 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:24:54.683 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:24:54.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:24:54.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:24:54.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:24:54.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:24:54.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:24:54.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:24:54.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:24:54.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:24:54.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:24:54.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:24:54.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:24:54.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:24:54.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:24:54.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:24:54.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:24:54.685 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 4b7b8b56-0151-49dc-aa8b-391396fbe19b 2024-02-05T05:24:54.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b7b8b56-0151-49dc-aa8b-391396fbe19b 2024-02-05T05:24:54.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4b7b8b56-0151-49dc-aa8b-391396fbe19b' 2024-02-05T05:24:54.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:24:54.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAmccBlgBAOKhAAm2AEn3QXiWDBH6XyB5RTuQ== 2024-02-05T05:24:54.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAmccBlgBAOKhAAm2AEn3QXiWDBH6XyB5RTuQ=="}' 2024-02-05T05:24:54.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b7b8b56-0151-49dc-aa8b-391396fbe19b -i td/osd-scrub-repair/1/new.json 2024-02-05T05:24:55.066 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:24:55.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:24:55.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAmccBlgBAOKhAAm2AEn3QXiWDBH6XyB5RTuQ== --osd-uuid 4b7b8b56-0151-49dc-aa8b-391396fbe19b 2024-02-05T05:24:55.113 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:55.113+0000 7fabceb91640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:55.168 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:55.169+0000 7fabceb91640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:55.169 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:55.169+0000 7fabceb91640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:55.170 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:55.169+0000 7fabceb91640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:24:55.170 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:55.169+0000 7fabceb91640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:24:55.170 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:55.169+0000 7fabceb91640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:24:55.238 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:24:55.237+0000 7fabceb91640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:25:02.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:25:02.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:25:02.054 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:25:02.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:25:02.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:25:02.430 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:25:02.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:25:02.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:25:02.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:25:02.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:25:02.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:25:02.457 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:02.457+0000 7f7c6166e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:02.459 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:02.457+0000 7f7c6166e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:02.460 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:02.461+0000 7f7c6166e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:02.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:25:02.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:25:02.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:25:02.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:25:02.773 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:25:02.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:25:02.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:02.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:25:02.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:02.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:03.001 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:03.001+0000 7f7c6166e640 -1 Falling back to public interface 2024-02-05T05:25:03.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:04.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:04.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:04.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:25:04.076 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:25:04.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:04.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:04.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:05.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:05.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:05.385 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:25:05.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:25:05.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:05.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:05.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:05.787 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:05.789+0000 7f7c6166e640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:25:06.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:06.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:06.752 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:25:06.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:25:06.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:06.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:07.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:08.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:08.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:08.358 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:25:08.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:25:08.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:08.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:08.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:09.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:09.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:09.764 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:25:09.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:25:09.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:09.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:10.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:11.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:11.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:11.164 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:25:11.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:25:11.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:11.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:11.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:12.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:12.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:12.635 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:25:12.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:25:12.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:12.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:12.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:14.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:14.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:14.001 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:25:14.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:25:14.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:14.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:14.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:15.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:15.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:15.303 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:25:15.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:25:15.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:15.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:15.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:16.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:16.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:16.944 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:25:16.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:25:16.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:16.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:17.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:18.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:18.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:18.261 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:25:18.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:25:18.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:18.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:18.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:19.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:19.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:19.564 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:25:19.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:25:19.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:19.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:19.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:20.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:20.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:20.939 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:25:20.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:25:20.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:20.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:22.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:23.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:23.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:23.010 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:25:23.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:25:23.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:23.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:23.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:24.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:24.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:24.895 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:25:24.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:25:24.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:24.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:25.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:26.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:26.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:26.673 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:25:26.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:25:26.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:26.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:27.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:28.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:28.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:28.729 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:25:28.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:25:28.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:28.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:29.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:30.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:30.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:30.835 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:25:30.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:25:30.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:30.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:31.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:32.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:32.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:32.192 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:25:32.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:25:32.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:32.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:25:32.493 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/267817872,v1:127.0.0.1:6811/267817872] [v2:127.0.0.1:6812/267817872,v1:127.0.0.1:6813/267817872] exists,up 4b7b8b56-0151-49dc-aa8b-391396fbe19b 2024-02-05T05:25:32.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:25:32.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:25:32.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:25:32.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:25:32.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-02-05T05:25:32.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:25:32.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:25:32.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:25:32.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:25:32.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:25:32.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:25:32.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:25:32.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:25:32.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:25:32.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:25:32.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:25:32.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:25:32.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:25:32.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:25:32.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:25:32.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:25:32.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:25:32.498 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:25:32.498 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:25:32.498 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:25:32.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:25:32.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:25:32.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:25:32.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:25:32.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:25:32.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:25:32.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:25:32.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:25:32.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:25:32.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:25:32.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:25:32.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:25:32.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:25:32.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:25:32.499 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:25:32.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f2ffc804-401e-4c2e-8f9e-8ee4bf3b4085 2024-02-05T05:25:32.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f2ffc804-401e-4c2e-8f9e-8ee4bf3b4085' 2024-02-05T05:25:32.500 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 f2ffc804-401e-4c2e-8f9e-8ee4bf3b4085 2024-02-05T05:25:32.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:25:32.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBMccBllP4hHxAAVd1NdBRfQL3PJAYiEePpWg== 2024-02-05T05:25:32.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBMccBllP4hHxAAVd1NdBRfQL3PJAYiEePpWg=="}' 2024-02-05T05:25:32.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f2ffc804-401e-4c2e-8f9e-8ee4bf3b4085 -i td/osd-scrub-repair/2/new.json 2024-02-05T05:25:32.931 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:25:32.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:25:32.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMccBllP4hHxAAVd1NdBRfQL3PJAYiEePpWg== --osd-uuid f2ffc804-401e-4c2e-8f9e-8ee4bf3b4085 2024-02-05T05:25:32.987 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:32.988+0000 7f30e7a05640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:33.033 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:33.032+0000 7f30e7a05640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:33.035 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:33.032+0000 7f30e7a05640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:33.035 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:33.036+0000 7f30e7a05640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:25:33.036 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:33.036+0000 7f30e7a05640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:25:33.036 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:33.036+0000 7f30e7a05640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:25:33.118 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:33.116+0000 7f30e7a05640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:25:40.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:25:40.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:25:40.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:25:40.183 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T05:25:40.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:25:40.564 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T05:25:40.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:25:40.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:25:40.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:25:40.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:25:40.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:25:40.596 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:40.596+0000 7fec76b31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:40.598 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:40.596+0000 7fec76b31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:40.600 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:40.600+0000 7fec76b31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:40.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:25:40.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:25:40.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:25:40.948 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:25:40.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:25:40.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:25:40.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:40.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:25:40.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:40.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:25:41.160 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:41.160+0000 7fec76b31640 -1 Falling back to public interface 2024-02-05T05:25:41.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:42.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:42.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:42.288 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:25:42.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:25:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:25:42.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:43.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:43.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:43.588 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:25:43.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:25:43.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:43.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:25:43.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:43.909 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:25:43.908+0000 7fec76b31640 -1 osd.2 0 log_to_monitors true 2024-02-05T05:25:44.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:44.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:44.910 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:25:44.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:25:44.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:44.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:25:45.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:46.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:46.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:46.379 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:25:46.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:25:46.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:46.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:25:46.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:47.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:47.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:47.698 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:25:47.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:25:47.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:47.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:25:48.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:49.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:49.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:49.121 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:25:49.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:25:49.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:49.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:25:49.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:50.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:50.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:50.441 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:25:50.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:25:50.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:50.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:25:50.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:51.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:51.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:51.942 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:25:51.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:25:51.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:51.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:25:52.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:53.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:53.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:53.263 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:25:53.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:25:53.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:53.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:25:53.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:54.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:54.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:54.578 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:25:54.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:25:54.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:54.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:25:55.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:56.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:56.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:56.092 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:25:56.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:25:56.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:56.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:25:56.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:57.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:57.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:57.413 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:25:57.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:25:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:25:57.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:58.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:58.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:58.917 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:25:58.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:25:58.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:58.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:25:59.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:00.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:00.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:00.860 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:26:00.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:26:00.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:00.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:26:01.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:02.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:02.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:02.281 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:26:02.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:26:02.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:02.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:26:02.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:03.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:03.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:03.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:26:03.789 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:26:03.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:03.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:26:04.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:05.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:05.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:05.307 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:26:05.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:26:05.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:05.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:26:05.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:06.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:06.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:06.852 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:26:06.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:26:06.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:06.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:26:07.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:08.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:08.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:08.735 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:26:08.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:26:08.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:08.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:26:09.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:10.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:10.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:10.220 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:26:10.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:26:10.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:10.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:26:10.533 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1433417908,v1:127.0.0.1:6819/1433417908] [v2:127.0.0.1:6820/1433417908,v1:127.0.0.1:6821/1433417908] exists,up f2ffc804-401e-4c2e-8f9e-8ee4bf3b4085 2024-02-05T05:26:10.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:26:10.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:26:10.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:26:10.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:26:10.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-02-05T05:26:10.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:26:10.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:26:10.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T05:26:10.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:26:10.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:26:10.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:26:10.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:26:10.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:26:10.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:26:10.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:26:10.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:26:10.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:26:10.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:26:10.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:26:10.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:26:10.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:26:10.537 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:26:10.537 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:26:10.537 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:26:10.537 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:26:10.537 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:26:10.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:26:10.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:26:10.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:26:10.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:26:10.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:26:10.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:26:10.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:26:10.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:26:10.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:26:10.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:26:10.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:26:10.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:26:10.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:26:10.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:26:10.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c45cee7e-c47e-4e9f-a0cb-f1770a36dec6 2024-02-05T05:26:10.539 INFO:tasks.workunit.client.0.smithi148.stdout:add osd3 c45cee7e-c47e-4e9f-a0cb-f1770a36dec6 2024-02-05T05:26:10.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 c45cee7e-c47e-4e9f-a0cb-f1770a36dec6' 2024-02-05T05:26:10.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:26:10.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQByccBlvseJIRAAQv8mhYrlZGEilEMQ/HpM5A== 2024-02-05T05:26:10.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQByccBlvseJIRAAQv8mhYrlZGEilEMQ/HpM5A=="}' 2024-02-05T05:26:10.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c45cee7e-c47e-4e9f-a0cb-f1770a36dec6 -i td/osd-scrub-repair/3/new.json 2024-02-05T05:26:10.945 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:26:10.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-02-05T05:26:10.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQByccBlvseJIRAAQv8mhYrlZGEilEMQ/HpM5A== --osd-uuid c45cee7e-c47e-4e9f-a0cb-f1770a36dec6 2024-02-05T05:26:10.999 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:26:10.996+0000 7f2c94670640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:11.047 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:26:11.048+0000 7f2c94670640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:11.049 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:26:11.048+0000 7f2c94670640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:11.050 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:26:11.048+0000 7f2c94670640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:26:11.050 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:26:11.048+0000 7f2c94670640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:26:11.050 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:26:11.048+0000 7f2c94670640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:26:11.126 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:26:11.124+0000 7f2c94670640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-02-05T05:26:17.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-02-05T05:26:17.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:26:17.806 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd3 key to auth repository 2024-02-05T05:26:17.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T05:26:17.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:26:18.193 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.3 2024-02-05T05:26:18.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T05:26:18.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:26:18.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:26:18.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:26:18.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:26:18.225 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:26:18.224+0000 7fdf81d64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:18.262 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:26:18.260+0000 7fdf81d64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:18.264 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:26:18.264+0000 7fdf81d64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:18.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T05:26:18.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:26:18.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:26:18.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:26:18.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:26:18.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:18.538 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:26:18.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:26:18.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:18.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:18.822 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:26:18.820+0000 7fdf81d64640 -1 Falling back to public interface 2024-02-05T05:26:18.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:19.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:19.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:19.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:26:19.862 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:26:19.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:19.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:20.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:21.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:21.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:21.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:26:21.166 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:26:21.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:21.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:21.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:21.589 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:26:21.587+0000 7fdf81d64640 -1 osd.3 0 log_to_monitors true 2024-02-05T05:26:22.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:22.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:22.564 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:26:22.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:26:22.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:22.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:22.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:23.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:23.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:23.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:26:23.949 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:26:23.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:23.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:24.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:25.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:25.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:25.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:26:25.306 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:26:25.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:25.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:25.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:26.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:26.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:26.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:26:26.701 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:26:26.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:26.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:27.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:28.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:28.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:28.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:26:28.021 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:26:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:28.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:29.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:29.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:29.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:26:29.334 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:26:29.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:29.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:29.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:30.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:30.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:30.699 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:26:30.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:26:30.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:30.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:31.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:32.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:32.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:32.008 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:26:32.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:26:32.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:32.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:32.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:33.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:33.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:33.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:26:33.326 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:26:33.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:33.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:33.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:34.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:34.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:34.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:26:34.710 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:26:34.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:34.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:35.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:36.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:36.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:36.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:26:36.041 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:26:36.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:36.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:36.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:37.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:37.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:37.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:26:37.497 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:26:37.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:37.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:38.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:39.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:39.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:39.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:26:39.344 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:26:39.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:39.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:40.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:41.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:41.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:41.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:26:41.183 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:26:41.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:41.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:41.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:42.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:42.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:42.712 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:26:42.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:26:42.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:42.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:43.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:44.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:44.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:44.723 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:26:44.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:26:44.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:44.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:45.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:46.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:46.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:46.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:26:46.213 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:26:46.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:46.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:46.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:47.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:47.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:47.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:26:47.813 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:26:47.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:47.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:26:48.141 INFO:tasks.workunit.client.0.smithi148.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/951097531,v1:127.0.0.1:6827/951097531] [v2:127.0.0.1:6828/951097531,v1:127.0.0.1:6829/951097531] exists,up c45cee7e-c47e-4e9f-a0cb-f1770a36dec6 2024-02-05T05:26:48.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:26:48.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:26:48.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:26:48.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-02-05T05:26:48.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:26:48.463 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T05:26:48.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:26:48.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:26:48.984 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T05:26:49.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:26:50.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:26:52.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-02-05T05:26:52.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:26:52.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:26:52.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:26:52.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:26:52.240 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:26:52.240 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:26:52.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:26:52.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:26:52.241 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:26:52.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:26:52.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:26:52.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:26:52.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:26:52.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:26:52.367 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:26:52.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:26:52.684 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:26:52.684 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:26:52.684 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T05:26:52.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:26:52.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:26:52.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-02-05T05:26:52.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-02-05T05:26:52.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-02-05T05:26:52.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:26:52.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-02-05T05:26:52.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-02-05T05:26:52.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672981' 2024-02-05T05:26:52.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:26:53.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-02-05T05:26:53.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-02-05T05:26:53.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672981 2-64424509458' 2024-02-05T05:26:53.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:53.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:26:53.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378629 2024-02-05T05:26:53.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378629 2024-02-05T05:26:53.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672981 2-64424509458 3-81604378629' 2024-02-05T05:26:53.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:53.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-02-05T05:26:53.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:53.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:26:53.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-02-05T05:26:53.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:53.263 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836509 2024-02-05T05:26:53.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-02-05T05:26:53.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-02-05T05:26:53.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:26:53.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-02-05T05:26:53.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:26:54.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:26:54.571 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:26:54.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836509 2024-02-05T05:26:54.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:54.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-02-05T05:26:54.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:54.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:26:54.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-02-05T05:26:54.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:54.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-02-05T05:26:54.890 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672981 2024-02-05T05:26:54.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-02-05T05:26:54.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:26:55.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-02-05T05:26:55.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:55.231 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-02-05T05:26:55.231 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:55.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:26:55.233 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-02-05T05:26:55.233 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:55.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-02-05T05:26:55.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-02-05T05:26:55.235 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509458 2024-02-05T05:26:55.235 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:26:55.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-02-05T05:26:55.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:55.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:55.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-81604378629 2024-02-05T05:26:55.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:26:55.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-81604378629 2024-02-05T05:26:55.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:55.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378629 2024-02-05T05:26:55.563 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.3 seq 81604378629 2024-02-05T05:26:55.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 81604378629' 2024-02-05T05:26:55.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:26:55.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378629 -lt 81604378629 2024-02-05T05:26:55.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:26:55.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:26:55.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:26:56.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:26:56.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:26:56.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:26:56.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:26:56.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:26:56.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:26:56.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:26:56.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:26:56.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:26:56.576 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:26:56.576 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:26:56.576 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:26:56.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:26:56.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:26:56.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:26:56.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-02-05T05:26:56.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:26:56.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:26:56.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-02-05T05:26:56.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:26:56.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-02-05T05:26:57.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:26:57.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:26:58.220 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'ecpool' created 2024-02-05T05:26:58.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:26:59.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-02-05T05:26:59.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:26:59.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:26:59.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:26:59.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:26:59.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:26:59.246 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:26:59.247 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:26:59.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:26:59.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:26:59.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:26:59.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:26:59.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:26:59.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:26:59.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:26:59.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:26:59.375 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:26:59.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:26:59.717 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:26:59.717 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:26:59.717 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T05:26:59.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:26:59.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:59.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:26:59.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-02-05T05:26:59.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-02-05T05:26:59.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-02-05T05:26:59.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:59.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:27:00.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-02-05T05:27:00.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-02-05T05:27:00.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672983' 2024-02-05T05:27:00.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:00.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:27:00.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-02-05T05:27:00.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-02-05T05:27:00.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672983 2-64424509460' 2024-02-05T05:27:00.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:00.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:27:00.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378631 2024-02-05T05:27:00.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378631 2024-02-05T05:27:00.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672983 2-64424509460 3-81604378631' 2024-02-05T05:27:00.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:00.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-02-05T05:27:00.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:00.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:27:00.301 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-02-05T05:27:00.301 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:00.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-02-05T05:27:00.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-02-05T05:27:00.302 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836512 2024-02-05T05:27:00.303 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:27:00.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836512 2024-02-05T05:27:00.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:27:01.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:27:01.623 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:27:01.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-02-05T05:27:01.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:01.939 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:01.939 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-02-05T05:27:01.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:27:01.941 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-02-05T05:27:01.941 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:01.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-02-05T05:27:01.943 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672983 2024-02-05T05:27:01.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-02-05T05:27:01.943 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:27:02.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-02-05T05:27:02.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:02.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-02-05T05:27:02.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:02.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:27:02.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-02-05T05:27:02.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:02.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-02-05T05:27:02.282 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509460 2024-02-05T05:27:02.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-02-05T05:27:02.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:27:02.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509460 2024-02-05T05:27:02.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:02.604 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-81604378631 2024-02-05T05:27:02.604 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:02.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:27:02.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-81604378631 2024-02-05T05:27:02.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:02.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378631 2024-02-05T05:27:02.609 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.3 seq 81604378631 2024-02-05T05:27:02.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 81604378631' 2024-02-05T05:27:02.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:27:03.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378631 -lt 81604378631 2024-02-05T05:27:03.040 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:27:03.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:03.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:03.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:27:03.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:27:03.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:27:03.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:27:03.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:27:03.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:27:03.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:27:03.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:27:03.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:27:03.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:27:03.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:03.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:04.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:27:04.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:27:04.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:27:04.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:27:04.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-02-05T05:27:04.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T05:27:04.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-02-05T05:27:04.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-02-05T05:27:04.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:27:04.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:27:04.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T05:27:04.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:27:04.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:27:04.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:27:04.649 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T05:27:04.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:27:05.151 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T05:27:05.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:27:05.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:27:05.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:27:05.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-02-05T05:27:05.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T05:27:05.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:27:05.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:27:05.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:27:05.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-02-05T05:27:05.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-02-05T05:27:05.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-02-05T05:27:05.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T05:27:05.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-02-05T05:27:05.821 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:27:05.821 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:27:06.120 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-02-05T05:27:06.120 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:27:06.120 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:27:06.120 INFO:tasks.workunit.client.0.smithi148.stderr:0' 2024-02-05T05:27:06.120 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-02-05T05:27:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-02-05T05:27:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-02-05T05:27:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-02-05T05:27:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-02-05T05:27:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-02-05T05:27:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:27:06.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:27:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-02-05T05:27:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:27:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:27:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:27:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:27:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:27:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:27:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:27:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:27:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:27:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:27:06.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:27:06.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:27:06.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:27:06.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:27:06.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:27:06.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:27:06.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:27:06.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:27:06.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:27:06.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:27:06.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:27:06.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:27:06.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:27:06.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:27:08.514 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:27:09.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:27:09.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:27:09.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:27:09.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:27:09.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:27:09.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:27:09.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:27:09.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:27:09.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:27:09.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:27:09.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:27:09.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:27:09.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:27:09.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:27:09.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:27:09.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:27:09.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:27:09.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:27:09.069 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:27:09.069 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:09.069 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:27:09.071 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:27:09.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:27:09.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:27:09.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:27:09.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:27:09.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:27:09.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:27:09.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:27:09.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:27:09.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:27:09.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:27:09.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:27:09.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:27:09.073 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.3 2024-02-05T05:27:09.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:27:09.073 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:27:09.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:27:09.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:27:09.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:27:09.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:27:09.104 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:09.102+0000 7fc2d5614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:09.107 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:09.106+0000 7fc2d5614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:09.109 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:09.106+0000 7fc2d5614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:09.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:27:09.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:27:09.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:27:09.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:27:09.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:27:09.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:09.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:27:09.400 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:27:09.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:09.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:09.658 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:09.654+0000 7fc2d5614640 -1 Falling back to public interface 2024-02-05T05:27:09.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:10.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:10.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:10.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:27:10.726 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:27:10.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:10.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:11.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:12.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:12.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:12.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:27:12.042 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:27:12.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:12.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:12.349 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:12.346+0000 7fc2d5614640 -1 osd.3 34 log_to_monitors true 2024-02-05T05:27:12.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:13.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:13.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:13.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:27:13.354 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:27:13.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:13.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:13.779 INFO:tasks.workunit.client.0.smithi148.stderr:osd.3 up in weight 1 up_from 39 up_thru 39 down_at 35 last_clean_interval [19,34) [v2:127.0.0.1:6826/2420290886,v1:127.0.0.1:6827/2420290886] [v2:127.0.0.1:6828/2420290886,v1:127.0.0.1:6829/2420290886] exists,up c45cee7e-c47e-4e9f-a0cb-f1770a36dec6 2024-02-05T05:27:13.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:27:13.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:27:13.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:27:13.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:27:13.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:27:13.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:27:13.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:27:13.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:27:13.780 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:27:13.781 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:27:13.781 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:27:13.781 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:27:13.781 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:27:13.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:27:13.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:27:13.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:27:13.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:27:13.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:27:13.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:27:14.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:27:14.232 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:27:14.232 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:27:14.232 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T05:27:14.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:27:14.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:14.233 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:27:14.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-02-05T05:27:14.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-02-05T05:27:14.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-02-05T05:27:14.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:14.507 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:27:14.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672987 2024-02-05T05:27:14.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672987 2024-02-05T05:27:14.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672987' 2024-02-05T05:27:14.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:14.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:27:14.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-02-05T05:27:14.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-02-05T05:27:14.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672987 2-64424509464' 2024-02-05T05:27:14.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:14.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:27:14.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-02-05T05:27:14.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-02-05T05:27:14.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672987 2-64424509464 3-167503724546' 2024-02-05T05:27:14.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:14.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-02-05T05:27:14.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:14.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:27:14.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-02-05T05:27:14.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:14.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-02-05T05:27:14.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-02-05T05:27:14.979 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836516 2024-02-05T05:27:14.980 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:27:15.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836516 2024-02-05T05:27:15.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:15.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672987 2024-02-05T05:27:15.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:15.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:27:15.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:15.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672987 2024-02-05T05:27:15.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672987 2024-02-05T05:27:15.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-02-05T05:27:15.288 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672987 2024-02-05T05:27:15.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:27:15.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672987 2024-02-05T05:27:15.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:15.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-02-05T05:27:15.623 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:15.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:27:15.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-02-05T05:27:15.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:15.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-02-05T05:27:15.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-02-05T05:27:15.626 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509464 2024-02-05T05:27:15.627 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:27:15.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509464 2024-02-05T05:27:15.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:27:16.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:27:16.963 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:27:17.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509464 2024-02-05T05:27:17.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:17.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-167503724546 2024-02-05T05:27:17.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:17.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:27:17.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-167503724546 2024-02-05T05:27:17.275 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:17.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-02-05T05:27:17.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 167503724546' 2024-02-05T05:27:17.276 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 167503724546 2024-02-05T05:27:17.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:27:17.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-02-05T05:27:17.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:27:17.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:17.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:17.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:27:17.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:27:17.970 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:27:17.970 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:27:17.970 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:27:17.970 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:27:17.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:27:17.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:27:18.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:27:18.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:27:18.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:18.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:27:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T05:27:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:27:18.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:27:18.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:27:18.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:27:18.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:27:18.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:27:18.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:27:18.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:27:18.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:27:18.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:27:18.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:27:18.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:27:19.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:27:19.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:27:19.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:19.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:19.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:27:19.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:27:19.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:27:19.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:27:19.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:27:19.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:27:19.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:27:19.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:27:19.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:27:19.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:27:19.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:27:19.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:27:19.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:27:19.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:27:19.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:27:19.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:27:20.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:27:20.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:27:20.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:27:20.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:27:20.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:27:20.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:27:20.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:27:20.303 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:27:20.304 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:20.304 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:20.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:27:20.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:27:20.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:27:20.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:27:20.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:27:20.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:27:20.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:27:20.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:27:20.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:27:21.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:27:21.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:27:21.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T05:27:21.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:27:21.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T05:27:21.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:27:21.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:27:21.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:27:21.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:27:21.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:27:21.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:27:21.480 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:27:21.480 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:27:21.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:27:21.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:27:21.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:21.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:22.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:27:22.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:27:22.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:27:22.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:27:22.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:27:22.145 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:27:22.145 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:27:22.145 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:27:22.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:27:22.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:27:22.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:27:22.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:27:22.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:22.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:22.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:22.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:22.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:26:58.147478+0000 2024-02-05T05:27:22.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:27:23.064 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:27:23.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:26:58.147478+0000 2024-02-05T05:27:23.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:27:23.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:26:58.147478+0000 2024-02-05T05:27:23.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:27:23.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:27:23.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:27:23.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:27:23.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:23.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:23.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:23.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:23.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:26:58.147478+0000 '>' 2024-02-05T05:26:58.147478+0000 2024-02-05T05:27:23.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:27:24.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:27:24.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:27:24.393 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:27:24.393 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:24.393 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:24.394 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:24.394 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:24.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:26:58.147478+0000 '>' 2024-02-05T05:26:58.147478+0000 2024-02-05T05:27:24.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:27:25.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:27:25.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:27:25.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:27:25.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:25.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:25.725 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:25.725 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:26.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:26:58.147478+0000 '>' 2024-02-05T05:26:58.147478+0000 2024-02-05T05:27:26.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:27:27.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:27:27.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:27:27.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:27:27.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:27.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:27.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:27.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:27.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:26:58.147478+0000 '>' 2024-02-05T05:26:58.147478+0000 2024-02-05T05:27:27.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:27:28.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:27:28.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:27:28.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:27:28.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:28.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:28.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:28.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:28.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:26:58.147478+0000 '>' 2024-02-05T05:26:58.147478+0000 2024-02-05T05:27:28.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:27:29.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:27:29.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:27:29.665 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:27:29.665 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:29.665 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:29.666 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:29.666 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:29.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:27:23.373106+0000 '>' 2024-02-05T05:26:58.147478+0000 2024-02-05T05:27:29.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:27:29.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:27:29.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:27:29.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:27:29.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:27:29.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:27:29.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:27:29.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:27:29.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:27:29.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:27:29.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:27:29.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:27:29.980 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:27:29.980 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:27:29.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:27:29.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:27:29.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:27:30.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:27:30.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:27:30.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:27:30.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:27:30.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:27:30.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:27:30.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:27:30.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:27:32.394 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:27:32.394 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:27:32.394 INFO:tasks.workunit.client.0.smithi148.stdout:hinfo_key 2024-02-05T05:27:32.394 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:27:32.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:27:32.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:27:32.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:27:32.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:27:32.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:27:32.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:27:32.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:27:32.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:27:32.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:27:32.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:27:32.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:27:32.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:27:32.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:27:32.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:27:32.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:27:32.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:27:32.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:27:32.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:27:32.695 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:27:32.695 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:32.695 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:27:32.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:27:32.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:27:32.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:27:32.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:27:32.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:27:32.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:27:32.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:27:32.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:27:32.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:27:32.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:27:32.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:27:32.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:27:32.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:27:32.697 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.3 2024-02-05T05:27:32.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:27:32.697 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:27:32.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:27:32.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:27:32.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:27:32.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:27:32.730 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:32.730+0000 7f6000b3f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:32.752 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:32.754+0000 7f6000b3f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:32.753 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:32.754+0000 7f6000b3f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:33.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:27:33.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:27:33.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:27:33.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:27:33.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:27:33.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:33.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:27:33.025 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:27:33.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:33.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:33.294 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:33.294+0000 7f6000b3f640 -1 Falling back to public interface 2024-02-05T05:27:33.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:34.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:34.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:34.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:27:34.390 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:27:34.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:34.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:34.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:35.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:35.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:35.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:27:35.705 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:27:35.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:35.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:35.974 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:35.973+0000 7f6000b3f640 -1 osd.3 42 log_to_monitors true 2024-02-05T05:27:36.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:37.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:37.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:37.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:27:37.016 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:27:37.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:37.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:37.426 INFO:tasks.workunit.client.0.smithi148.stderr:osd.3 up in weight 1 up_from 46 up_thru 46 down_at 42 last_clean_interval [39,42) [v2:127.0.0.1:6826/2668755362,v1:127.0.0.1:6827/2668755362] [v2:127.0.0.1:6828/2668755362,v1:127.0.0.1:6829/2668755362] exists,up c45cee7e-c47e-4e9f-a0cb-f1770a36dec6 2024-02-05T05:27:37.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:27:37.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:27:37.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:27:37.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:27:37.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:27:37.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:27:37.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:27:37.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:27:37.427 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:27:37.427 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:27:37.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:27:37.428 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:27:37.428 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:27:37.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:27:37.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:27:37.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:27:37.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:27:37.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:27:37.541 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:27:37.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:27:37.891 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:27:37.891 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:27:37.891 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T05:27:37.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:27:37.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:37.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:27:38.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836521 2024-02-05T05:27:38.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836521 2024-02-05T05:27:38.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521' 2024-02-05T05:27:38.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:38.037 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:27:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672993 2024-02-05T05:27:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672993 2024-02-05T05:27:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-42949672993' 2024-02-05T05:27:38.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:38.275 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:27:38.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509470 2024-02-05T05:27:38.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509470 2024-02-05T05:27:38.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-42949672993 2-64424509470' 2024-02-05T05:27:38.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:38.419 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:27:38.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495618 2024-02-05T05:27:38.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495618 2024-02-05T05:27:38.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-42949672993 2-64424509470 3-197568495618' 2024-02-05T05:27:38.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:38.552 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836521 2024-02-05T05:27:38.552 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:38.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:27:38.553 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836521 2024-02-05T05:27:38.553 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:38.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836521 2024-02-05T05:27:38.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-02-05T05:27:38.554 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836521 2024-02-05T05:27:38.555 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:27:38.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836521 -lt 21474836521 2024-02-05T05:27:38.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:38.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:38.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672993 2024-02-05T05:27:38.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:27:38.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672993 2024-02-05T05:27:38.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:38.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672993 2024-02-05T05:27:38.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-02-05T05:27:38.845 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672993 2024-02-05T05:27:38.845 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:27:39.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672993 -lt 42949672993 2024-02-05T05:27:39.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:39.164 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509470 2024-02-05T05:27:39.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:39.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:27:39.167 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509470 2024-02-05T05:27:39.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:39.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509470 2024-02-05T05:27:39.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-02-05T05:27:39.169 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509470 2024-02-05T05:27:39.169 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:27:39.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509470 -lt 64424509470 2024-02-05T05:27:39.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:39.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495618 2024-02-05T05:27:39.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:39.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:27:39.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495618 2024-02-05T05:27:39.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:39.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495618 2024-02-05T05:27:39.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495618' 2024-02-05T05:27:39.496 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 197568495618 2024-02-05T05:27:39.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:27:39.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495618 -lt 197568495618 2024-02-05T05:27:39.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:27:39.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:39.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:40.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:27:40.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:27:40.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:27:40.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:27:40.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:27:40.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:27:40.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:27:40.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:27:40.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:27:40.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:27:40.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:40.499 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:40.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:27:40.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:27:40.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:27:40.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:27:40.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:27:40.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-02-05T05:27:40.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:27:40.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:27:40.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-02-05T05:27:40.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:27:40.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:27:40.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:27:40.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:27:40.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:27:40.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:27:40.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:27:40.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:27:40.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:27:40.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:27:40.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:27:40.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:27:40.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:27:40.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:27:40.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:27:40.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:27:41.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:27:41.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:27:41.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:27:41.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:27:41.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:27:41.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:27:41.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:27:41.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:27:43.021 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:27:43.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:27:43.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:27:43.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:27:43.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:27:43.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:27:43.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:27:43.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:27:43.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:27:43.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:27:43.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:27:43.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:27:43.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:27:43.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:27:43.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:27:43.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:27:43.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:27:43.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:27:43.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:27:43.561 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:27:43.561 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:43.561 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:27:43.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:27:43.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:27:43.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:27:43.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:27:43.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:27:43.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:27:43.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:27:43.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:27:43.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:27:43.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:27:43.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:27:43.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:27:43.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:27:43.562 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:27:43.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:27:43.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:27:43.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:27:43.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:27:43.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:27:43.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:27:43.596 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:43.597+0000 7f7b06cbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:43.598 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:43.597+0000 7f7b06cbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:43.600 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:43.601+0000 7f7b06cbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:43.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:27:43.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:27:43.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:27:43.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:27:43.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:27:43.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:43.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:27:43.886 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:27:43.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:43.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:27:44.142 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:44.141+0000 7f7b06cbb640 -1 Falling back to public interface 2024-02-05T05:27:44.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:45.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:45.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:45.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:27:45.208 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:27:45.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:45.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:27:45.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:46.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:46.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:46.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:27:46.518 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:27:46.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:46.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:27:46.677 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:46.677+0000 7f7b06cbb640 -1 osd.1 48 log_to_monitors true 2024-02-05T05:27:46.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:47.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:47.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:47.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:27:47.838 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:27:47.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:47.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:27:48.441 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 52 up_thru 52 down_at 49 last_clean_interval [10,48) [v2:127.0.0.1:6810/1029985552,v1:127.0.0.1:6811/1029985552] [v2:127.0.0.1:6812/1029985552,v1:127.0.0.1:6813/1029985552] exists,up 4b7b8b56-0151-49dc-aa8b-391396fbe19b 2024-02-05T05:27:48.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:27:48.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:27:48.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:27:48.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:27:48.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:27:48.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:27:48.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:27:48.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:27:48.443 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:27:48.443 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:27:48.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:27:48.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:27:48.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:27:48.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:27:48.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:27:48.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:27:48.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:27:48.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:27:48.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:27:48.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:27:48.872 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:27:48.872 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:27:48.872 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T05:27:48.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:27:48.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:48.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:27:49.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836524 2024-02-05T05:27:49.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836524 2024-02-05T05:27:49.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524' 2024-02-05T05:27:49.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:49.022 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:27:49.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299394 2024-02-05T05:27:49.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299394 2024-02-05T05:27:49.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-223338299394' 2024-02-05T05:27:49.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:49.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:27:49.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509473 2024-02-05T05:27:49.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509473 2024-02-05T05:27:49.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-223338299394 2-64424509473' 2024-02-05T05:27:49.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:49.303 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:27:49.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495621 2024-02-05T05:27:49.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495621 2024-02-05T05:27:49.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-223338299394 2-64424509473 3-197568495621' 2024-02-05T05:27:49.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:49.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836524 2024-02-05T05:27:49.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:49.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:27:49.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836524 2024-02-05T05:27:49.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:49.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836524 2024-02-05T05:27:49.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-02-05T05:27:49.482 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836524 2024-02-05T05:27:49.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:27:49.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836524 2024-02-05T05:27:49.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:27:50.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:27:50.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:27:51.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836524 2024-02-05T05:27:51.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:51.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:51.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-223338299394 2024-02-05T05:27:51.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:27:51.137 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-223338299394 2024-02-05T05:27:51.137 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:51.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299394 2024-02-05T05:27:51.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 223338299394' 2024-02-05T05:27:51.138 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 223338299394 2024-02-05T05:27:51.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:27:51.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299394 -lt 223338299394 2024-02-05T05:27:51.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:51.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:51.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509473 2024-02-05T05:27:51.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:27:51.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509473 2024-02-05T05:27:51.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:51.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509473 2024-02-05T05:27:51.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-02-05T05:27:51.496 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509473 2024-02-05T05:27:51.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:27:51.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509474 -lt 64424509473 2024-02-05T05:27:51.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:51.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495621 2024-02-05T05:27:51.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:51.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:27:51.833 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:51.833 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495621 2024-02-05T05:27:51.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495621 2024-02-05T05:27:51.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495621' 2024-02-05T05:27:51.834 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 197568495621 2024-02-05T05:27:51.834 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:27:52.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495621 -lt 197568495621 2024-02-05T05:27:52.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:27:52.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:52.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:52.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:27:52.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:27:52.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:27:52.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:27:52.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:27:52.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:27:52.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:27:52.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:27:52.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:27:52.854 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:27:52.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:52.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:53.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:27:53.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:27:53.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:27:53.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:27:53.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:27:53.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:27:53.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:27:53.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:27:53.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:27:53.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:27:53.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:27:53.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:27:53.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:53.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:53.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:53.544 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:53.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:27:23.373106+0000 2024-02-05T05:27:53.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:27:54.129 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:27:54.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:27:23.373106+0000 2024-02-05T05:27:54.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:27:54.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:27:23.373106+0000 2024-02-05T05:27:54.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:27:54.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:27:54.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:27:54.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:27:54.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:54.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:54.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:54.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:54.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:27:23.373106+0000 '>' 2024-02-05T05:27:23.373106+0000 2024-02-05T05:27:54.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:27:55.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:27:55.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:27:55.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:27:55.461 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:55.461 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:55.461 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:55.461 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:55.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:27:23.373106+0000 '>' 2024-02-05T05:27:23.373106+0000 2024-02-05T05:27:55.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:27:56.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:27:56.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:27:56.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:27:56.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:56.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:56.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:56.781 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:57.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:27:54.250876+0000 '>' 2024-02-05T05:27:23.373106+0000 2024-02-05T05:27:57.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:27:57.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:27:57.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:27:57.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:27:57.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:27:57.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:27:57.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:27:57.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:27:57.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:27:57.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:27:57.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:27:57.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:27:57.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:27:57.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:27:57.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:27:57.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:27:57.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:27:57.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:27:57.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:27:57.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:27:57.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:27:57.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:27:57.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:27:57.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:27:57.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:27:59.063 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:27:59.063 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:27:59.063 INFO:tasks.workunit.client.0.smithi148.stdout:hinfo_key 2024-02-05T05:27:59.063 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:27:59.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:27:59.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:27:59.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:27:59.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:27:59.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:27:59.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:27:59.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:27:59.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:27:59.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:27:59.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:27:59.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:27:59.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:27:59.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:27:59.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:27:59.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:27:59.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:27:59.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:27:59.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:27:59.352 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:27:59.352 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:59.352 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:27:59.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:27:59.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:27:59.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:27:59.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:27:59.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:27:59.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:27:59.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:27:59.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:27:59.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:27:59.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:27:59.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:27:59.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:27:59.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:27:59.354 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.3 2024-02-05T05:27:59.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:27:59.354 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:27:59.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:27:59.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:27:59.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:27:59.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:27:59.388 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:59.389+0000 7f3b6c0fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:59.391 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:59.389+0000 7f3b6c0fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:59.393 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:59.393+0000 7f3b6c0fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:59.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:27:59.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:27:59.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:27:59.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:27:59.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:27:59.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:59.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:27:59.672 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:27:59.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:59.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:59.925 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:27:59.925+0000 7f3b6c0fb640 -1 Falling back to public interface 2024-02-05T05:28:00.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:01.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:01.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:01.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:28:01.009 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:28:01.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:01.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:28:01.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:02.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:02.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:02.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:28:02.328 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:28:02.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:02.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:28:02.330 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:28:02.329+0000 7f3b6c0fb640 -1 osd.3 53 log_to_monitors true 2024-02-05T05:28:02.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:03.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:03.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:03.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:28:03.741 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:28:03.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:03.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:28:04.208 INFO:tasks.workunit.client.0.smithi148.stderr:osd.3 up in weight 1 up_from 57 up_thru 57 down_at 54 last_clean_interval [46,53) [v2:127.0.0.1:6826/2217980153,v1:127.0.0.1:6827/2217980153] [v2:127.0.0.1:6828/2217980153,v1:127.0.0.1:6829/2217980153] exists,up c45cee7e-c47e-4e9f-a0cb-f1770a36dec6 2024-02-05T05:28:04.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:28:04.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:28:04.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:28:04.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:28:04.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:28:04.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:28:04.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:28:04.209 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:28:04.210 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:28:04.210 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:28:04.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:28:04.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:28:04.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:28:04.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:28:04.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:28:04.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:28:04.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:28:04.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:28:04.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:28:04.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:28:04.644 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:28:04.644 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:28:04.644 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T05:28:04.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:28:04.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:04.644 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:28:04.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836529 2024-02-05T05:28:04.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836529 2024-02-05T05:28:04.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529' 2024-02-05T05:28:04.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:04.852 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:28:04.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299398 2024-02-05T05:28:04.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299398 2024-02-05T05:28:04.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-223338299398' 2024-02-05T05:28:04.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:04.994 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:28:05.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509477 2024-02-05T05:28:05.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509477 2024-02-05T05:28:05.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-223338299398 2-64424509477' 2024-02-05T05:28:05.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:05.132 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:28:05.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135874 2024-02-05T05:28:05.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135874 2024-02-05T05:28:05.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-223338299398 2-64424509477 3-244813135874' 2024-02-05T05:28:05.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:05.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836529 2024-02-05T05:28:05.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:05.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:28:05.275 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836529 2024-02-05T05:28:05.275 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:05.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836529 2024-02-05T05:28:05.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-02-05T05:28:05.276 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836529 2024-02-05T05:28:05.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:05.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836529 2024-02-05T05:28:05.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:28:06.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:28:06.573 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:06.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836529 2024-02-05T05:28:06.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:06.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:06.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-223338299398 2024-02-05T05:28:06.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:28:06.875 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-223338299398 2024-02-05T05:28:06.875 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:06.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299398 2024-02-05T05:28:06.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 223338299398' 2024-02-05T05:28:06.876 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 223338299398 2024-02-05T05:28:06.876 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:28:07.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299398 -lt 223338299398 2024-02-05T05:28:07.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:07.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509477 2024-02-05T05:28:07.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:07.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:28:07.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509477 2024-02-05T05:28:07.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:07.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509477 2024-02-05T05:28:07.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509477' 2024-02-05T05:28:07.190 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509477 2024-02-05T05:28:07.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:28:07.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509478 -lt 64424509477 2024-02-05T05:28:07.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:07.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-244813135874 2024-02-05T05:28:07.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:07.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:28:07.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-244813135874 2024-02-05T05:28:07.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:07.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135874 2024-02-05T05:28:07.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 244813135874' 2024-02-05T05:28:07.522 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 244813135874 2024-02-05T05:28:07.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:28:07.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135874 -lt 244813135874 2024-02-05T05:28:07.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:28:07.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:07.808 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:08.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:28:08.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:28:08.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:28:08.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:28:08.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:28:08.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:28:08.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:28:08.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:28:08.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:28:08.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:28:08.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:08.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:08.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:28:08.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:28:08.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:28:08.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:28:08.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:28:08.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-02-05T05:28:08.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:28:08.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:28:08.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-02-05T05:28:08.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-02-05T05:28:08.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:28:08.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:28:08.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:28:08.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:28:08.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 157785"' 2024-02-05T05:28:08.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 157785' 2024-02-05T05:28:08.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:28:08.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:28:08.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:28:08.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 157787"' 2024-02-05T05:28:08.955 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 157787' 2024-02-05T05:28:08.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:28:08.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/157788: /' 2024-02-05T05:28:08.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 157785 157787' 2024-02-05T05:28:08.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:28:08.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:28:08.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 157785 2024-02-05T05:28:08.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:28:08.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:28:08.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/157790: /' 2024-02-05T05:28:12.276 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:28:12.276 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:28:12.277 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:28:12.277 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:28:12.277 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:28:12.277 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:28:12.277 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:28:12.277 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:28:12.277 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:28:12.277 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:28:12.277 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:28:12.277 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:28:12.277 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:28:12.277 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:28:12.277 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:28:12.278 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:28:12.278 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:28:12.278 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:28:12.278 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:28:12.278 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:28:12.278 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:28:12.278 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:28:12.278 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:28:12.278 INFO:tasks.workunit.client.0.smithi148.stderr:157788: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:28:12.278 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:28:12.278 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:28:12.278 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:28:12.278 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:28:12.278 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:28:12.279 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:28:12.279 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:28:12.279 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:28:12.281 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:28:12.281 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:28:12.281 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:28:12.281 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:28:12.281 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:28:12.281 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:28:12.281 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:28:12.281 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:28:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:28:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:28:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:157788: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:28:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:157788: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:28:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:157788: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:28:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:28:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:28:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:28:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:28:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:28:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:28:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:28:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:28:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:28:12.283 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:28:12.283 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:28:12.283 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:28:12.283 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:28:12.283 INFO:tasks.workunit.client.0.smithi148.stderr:157788: start osd.1 2024-02-05T05:28:12.283 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:28:12.411 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:28:12.412 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:28:12.412 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T05:28:12.412 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:28:12.412 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:28:12.412 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:28:12.412 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:28:12.412 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T05:28:12.412 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:28:12.412 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:28:12.412 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:28:12.412 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:28:12.412 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:28:12.413 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:28:12.413 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:28:12.413 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:28:12.413 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:28:12.413 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:28:12.413 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:28:12.413 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T05:28:12.413 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:28:12.413 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:28:12.413 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-02-05T05:28:12.413 INFO:tasks.workunit.client.0.smithi148.stderr:157790: remove 2#2:eb822e21:::SOMETHING:head# 2024-02-05T05:28:12.413 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T05:28:12.413 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:28:12.413 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:28:12.414 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:28:12.414 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:28:12.414 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:28:12.414 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:28:12.414 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:28:12.414 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:28:12.414 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:28:12.414 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:28:12.414 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:28:12.414 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:28:12.415 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:28:12.415 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:28:12.415 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:28:12.415 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:28:12.415 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:28:12.415 INFO:tasks.workunit.client.0.smithi148.stderr:157790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:28:12.415 INFO:tasks.workunit.client.0.smithi148.stderr:157790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:28:12.415 INFO:tasks.workunit.client.0.smithi148.stderr:157790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:28:12.415 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:28:12.415 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:28:12.415 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:28:12.415 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:28:12.415 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:28:12.415 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:28:12.416 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:28:12.416 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:28:12.416 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:28:12.416 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:28:12.416 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:28:12.416 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:28:12.416 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:28:12.416 INFO:tasks.workunit.client.0.smithi148.stderr:157790: start osd.2 2024-02-05T05:28:12.416 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:28:16.925 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:28:16.925 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:28:16.925 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:28:16.925 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:28:16.925 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:28:16.926 INFO:tasks.workunit.client.0.smithi148.stderr:157788: 2024-02-05T05:28:12.312+0000 7f6214260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:16.926 INFO:tasks.workunit.client.0.smithi148.stderr:157788: 2024-02-05T05:28:12.316+0000 7f6214260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:16.926 INFO:tasks.workunit.client.0.smithi148.stderr:157788: 2024-02-05T05:28:12.316+0000 7f6214260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:16.926 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:28:16.926 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:28:16.926 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:28:16.926 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:28:16.926 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:28:16.926 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:16.926 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:28:16.926 INFO:tasks.workunit.client.0.smithi148.stderr:157788: 0 2024-02-05T05:28:16.927 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:16.927 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:16.927 INFO:tasks.workunit.client.0.smithi148.stderr:157788: 2024-02-05T05:28:12.880+0000 7f6214260640 -1 Falling back to public interface 2024-02-05T05:28:16.927 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:16.927 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:16.927 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:16.927 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:28:16.927 INFO:tasks.workunit.client.0.smithi148.stderr:157788: 1 2024-02-05T05:28:16.927 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:16.927 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:16.927 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:16.928 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:16.928 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:16.928 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:28:16.928 INFO:tasks.workunit.client.0.smithi148.stderr:157788: 2 2024-02-05T05:28:16.928 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:16.928 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:16.928 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:16.928 INFO:tasks.workunit.client.0.smithi148.stderr:157788: 2024-02-05T05:28:15.888+0000 7f6214260640 -1 osd.1 60 log_to_monitors true 2024-02-05T05:28:16.928 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:16.928 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:16.928 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:28:16.928 INFO:tasks.workunit.client.0.smithi148.stderr:157788: 3 2024-02-05T05:28:16.929 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:16.929 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:17.426 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:28:17.426 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:28:17.426 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:28:17.426 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:28:17.426 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:28:17.426 INFO:tasks.workunit.client.0.smithi148.stderr:157790: 2024-02-05T05:28:12.440+0000 7fe4754fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:17.426 INFO:tasks.workunit.client.0.smithi148.stderr:157790: 2024-02-05T05:28:12.444+0000 7fe4754fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:17.426 INFO:tasks.workunit.client.0.smithi148.stderr:157790: 2024-02-05T05:28:12.444+0000 7fe4754fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:17.426 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:28:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:28:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:28:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:28:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:28:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:28:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:157790: 0 2024-02-05T05:28:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:28:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:157790: 2024-02-05T05:28:13.016+0000 7fe4754fe640 -1 Falling back to public interface 2024-02-05T05:28:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:28:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:157790: 1 2024-02-05T05:28:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:28:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:28:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:157790: 2 2024-02-05T05:28:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:28:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:157790: 2024-02-05T05:28:16.296+0000 7fe4754fe640 -1 osd.2 59 log_to_monitors true 2024-02-05T05:28:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:17.429 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:28:17.429 INFO:tasks.workunit.client.0.smithi148.stderr:157790: 3 2024-02-05T05:28:17.429 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:17.429 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:28:19.229 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:19.230 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:19.230 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:19.230 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:28:19.230 INFO:tasks.workunit.client.0.smithi148.stderr:157788: 4 2024-02-05T05:28:19.230 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:19.230 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:19.230 INFO:tasks.workunit.client.0.smithi148.stderr:157788: osd.1 up in weight 1 up_from 65 up_thru 65 down_at 60 last_clean_interval [52,60) [v2:127.0.0.1:6810/3314032419,v1:127.0.0.1:6811/3314032419] [v2:127.0.0.1:6812/3314032419,v1:127.0.0.1:6813/3314032419] exists,up 4b7b8b56-0151-49dc-aa8b-391396fbe19b 2024-02-05T05:28:19.230 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:28:19.230 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:28:19.231 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:28:19.231 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:28:19.231 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:28:19.231 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:28:19.231 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:28:19.231 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:28:19.231 INFO:tasks.workunit.client.0.smithi148.stderr:157788: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:28:19.231 INFO:tasks.workunit.client.0.smithi148.stderr:157788: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:28:19.231 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:28:19.231 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:28:19.231 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:28:19.231 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:28:19.231 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:28:19.232 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:28:19.232 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:28:19.232 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:28:19.232 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:28:19.232 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:28:19.232 INFO:tasks.workunit.client.0.smithi148.stderr:157788: 1 2024-02-05T05:28:19.232 INFO:tasks.workunit.client.0.smithi148.stderr:157788: 2 2024-02-05T05:28:19.232 INFO:tasks.workunit.client.0.smithi148.stderr:157788: 3' 2024-02-05T05:28:19.232 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:28:19.232 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:19.232 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:28:19.233 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836532 2024-02-05T05:28:19.233 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836532 2024-02-05T05:28:19.233 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532' 2024-02-05T05:28:19.233 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:19.743 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hels.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:19.743 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:19.743 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:19.743 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:28:19.743 INFO:tasks.workunit.client.0.smithi148.stderr:157790: 4 2024-02-05T05:28:19.743 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:19.743 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:28:19.743 INFO:tasks.workunit.client.0.smithi148.stderr:157790: osd.2 up in weight 1 up_from 66 up_thru 49 down_at 60 last_clean_interval [15,59) [v2:127.0.0.1:6818/4073881059,v1:127.0.0.1:6819/4073881059] [v2:127.0.0.1:6820/4073881059,v1:127.0.0.1:6821/4073881059] exists,up f2ffc804-401e-4c2e-8f9e-8ee4bf3b4085 2024-02-05T05:28:19.744 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:28:19.744 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:28:19.744 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:28:19.744 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:28:19.744 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:28:19.744 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:28:19.744 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:28:19.744 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:28:19.744 INFO:tasks.workunit.client.0.smithi148.stderr:157790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:28:19.744 INFO:tasks.workunit.client.0.smithi148.stderr:157790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:28:19.744 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:28:19.744 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:28:19.744 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:28:19.744 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:28:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:28:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:28:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:28:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:28:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:28:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:28:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:157790: 1 2024-02-05T05:28:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:157790: 2 2024-02-05T05:28:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:157790: 3' 2024-02-05T05:28:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:28:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:28:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836534 2024-02-05T05:28:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836534 2024-02-05T05:28:19.746 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534' 2024-02-05T05:28:19.746 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:21.402 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:28:21.402 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874242 2024-02-05T05:28:21.402 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874242 2024-02-05T05:28:21.402 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-279172874242' 2024-02-05T05:28:21.402 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:21.402 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:28:21.402 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-02-05T05:28:21.402 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-02-05T05:28:21.402 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-279172874242 2-283467841538' 2024-02-05T05:28:21.402 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:21.403 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:28:21.403 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135878 2024-02-05T05:28:21.403 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135878 2024-02-05T05:28:21.403 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-279172874242 2-283467841538 3-244813135878' 2024-02-05T05:28:21.403 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:21.403 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836532 2024-02-05T05:28:21.403 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:21.403 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:28:21.403 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836532 2024-02-05T05:28:21.403 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:21.403 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836532 2024-02-05T05:28:21.403 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-02-05T05:28:21.403 INFO:tasks.workunit.client.0.smithi148.stderr:157788: waiting osd.0 seq 21474836532 2024-02-05T05:28:21.404 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:21.404 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836532 2024-02-05T05:28:21.404 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:28:21.404 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:28:21.404 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:21.404 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836532 2024-02-05T05:28:21.404 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:21.404 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874242 2024-02-05T05:28:21.404 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:21.404 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:28:21.404 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874242 2024-02-05T05:28:21.404 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:21.764 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/upers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:28:21.765 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874243 2024-02-05T05:28:21.765 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874243 2024-02-05T05:28:21.765 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-279172874243' 2024-02-05T05:28:21.765 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:21.765 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:28:21.765 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841539 2024-02-05T05:28:21.765 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841539 2024-02-05T05:28:21.765 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-279172874243 2-283467841539' 2024-02-05T05:28:21.765 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:21.765 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:28:21.765 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135879 2024-02-05T05:28:21.765 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135879 2024-02-05T05:28:21.765 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-279172874243 2-283467841539 3-244813135879' 2024-02-05T05:28:21.765 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:21.766 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836534 2024-02-05T05:28:21.766 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:21.766 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:28:21.766 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836534 2024-02-05T05:28:21.766 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:21.766 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836534 2024-02-05T05:28:21.766 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-02-05T05:28:21.766 INFO:tasks.workunit.client.0.smithi148.stderr:157790: waiting osd.0 seq 21474836534 2024-02-05T05:28:21.766 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:21.766 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836534 2024-02-05T05:28:21.766 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:28:21.766 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:28:21.766 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:21.766 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836534 2024-02-05T05:28:21.767 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:21.767 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874243 2024-02-05T05:28:21.767 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:21.767 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:28:21.767 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874243 2024-02-05T05:28:21.767 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:22.825 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874242 2024-02-05T05:28:22.825 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874242' 2024-02-05T05:28:22.825 INFO:tasks.workunit.client.0.smithi148.stderr:157788: waiting osd.1 seq 279172874242 2024-02-05T05:28:22.825 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:28:22.825 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874243 -lt 279172874242 2024-02-05T05:28:22.825 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:22.826 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-283467841538 2024-02-05T05:28:22.826 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:22.826 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:28:22.826 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-283467841538 2024-02-05T05:28:22.826 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:22.826 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-02-05T05:28:22.826 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 283467841538' 2024-02-05T05:28:22.826 INFO:tasks.workunit.client.0.smithi148.stderr:157788: waiting osd.2 seq 283467841538 2024-02-05T05:28:22.826 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:28:22.826 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841538 2024-02-05T05:28:22.826 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:22.826 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-244813135878 2024-02-05T05:28:22.826 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:22.826 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:28:22.827 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-244813135878 2024-02-05T05:28:22.827 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:22.827 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135878 2024-02-05T05:28:22.827 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 244813135878' 2024-02-05T05:28:22.827 INFO:tasks.workunit.client.0.smithi148.stderr:157788: waiting osd.3 seq 244813135878 2024-02-05T05:28:22.827 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:28:22.827 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135879 -lt 244813135878 2024-02-05T05:28:22.827 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:28:22.827 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:22.827 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:22.827 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:28:22.827 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:28:22.827 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:28:22.827 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:28:22.828 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:28:23.132 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874243 2024-02-05T05:28:23.133 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874243' 2024-02-05T05:28:23.133 INFO:tasks.workunit.client.0.smithi148.stderr:157790: waiting osd.1 seq 279172874243 2024-02-05T05:28:23.133 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:28:23.133 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874243 -lt 279172874243 2024-02-05T05:28:23.133 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:23.133 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-283467841539 2024-02-05T05:28:23.133 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:23.133 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:28:23.133 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-283467841539 2024-02-05T05:28:23.133 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:23.133 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841539 2024-02-05T05:28:23.133 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 283467841539' 2024-02-05T05:28:23.133 INFO:tasks.workunit.client.0.smithi148.stderr:157790: waiting osd.2 seq 283467841539 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841539 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-244813135879 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-244813135879 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135879 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 244813135879' 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: waiting osd.3 seq 244813135879 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135879 -lt 244813135879 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:28:23.134 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:23.135 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:23.135 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:28:23.135 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:28:23.135 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:28:23.135 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:28:23.135 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:28:23.509 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:28:23.509 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:28:23.509 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:28:23.509 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:28:23.509 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:28:23.509 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:23.510 INFO:tasks.workunit.client.0.smithi148.stderr:157788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:23.510 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:28:23.510 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:28:23.510 INFO:tasks.workunit.client.0.smithi148.stderr:157788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:28:23.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:28:23.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:28:23.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 157787 2024-02-05T05:28:23.814 INFO:tasks.workunit.client.0.smithi148.stderr:stale") | not)' 2024-02-05T05:28:23.814 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:28:23.814 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:28:23.815 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:28:23.815 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:28:23.815 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:23.815 INFO:tasks.workunit.client.0.smithi148.stderr:157790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:23.815 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:28:23.815 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:28:23.815 INFO:tasks.workunit.client.0.smithi148.stderr:157790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:28:23.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:28:23.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:28:23.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:28:23.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:28:23.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:28:23.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:28:23.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:28:23.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:28:23.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:28:23.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:28:23.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:28:24.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:28:24.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:28:24.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:28:24.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:28:24.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:28:24.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:28:24.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:28:24.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:28:24.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:27:54.250876+0000 2024-02-05T05:28:24.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:28:24.730 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:28:24.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:27:54.250876+0000 2024-02-05T05:28:24.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:28:24.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:27:54.250876+0000 2024-02-05T05:28:24.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:28:24.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:28:24.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:28:24.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:28:24.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:28:24.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:28:24.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:28:24.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:28:25.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:27:54.250876+0000 '>' 2024-02-05T05:27:54.250876+0000 2024-02-05T05:28:25.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:28:26.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:28:26.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:28:26.060 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:28:26.060 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:28:26.060 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:28:26.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:28:26.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:28:26.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:27:54.250876+0000 '>' 2024-02-05T05:27:54.250876+0000 2024-02-05T05:28:26.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:28:27.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:28:27.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:28:27.370 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:28:27.370 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:28:27.370 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:28:27.371 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:28:27.371 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:28:27.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:27:54.250876+0000 '>' 2024-02-05T05:27:54.250876+0000 2024-02-05T05:28:27.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:28:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:28:28.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:28:28.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:28:28.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:28:28.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:28:28.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:28:28.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:28:28.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:25.579903+0000 '>' 2024-02-05T05:27:54.250876+0000 2024-02-05T05:28:28.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:28:28.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:28:28.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:28:28.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:28:28.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:28:28.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 160660"' 2024-02-05T05:28:28.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 160660' 2024-02-05T05:28:28.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:28:28.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:28:28.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:28:28.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:28:28.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/160662: /' 2024-02-05T05:28:28.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 160663"' 2024-02-05T05:28:28.992 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 160663' 2024-02-05T05:28:28.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:28:28.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 160660 160663' 2024-02-05T05:28:28.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:28:28.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:28:28.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 160660 2024-02-05T05:28:28.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/160667: /' 2024-02-05T05:28:28.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:28:31.986 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:28:31.986 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:28:31.986 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T05:28:31.986 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:28:31.986 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:28:31.986 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:28:31.986 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:28:31.986 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T05:28:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:28:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:28:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:28:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:28:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:28:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:28:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:28:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:28:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:28:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:28:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:28:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T05:28:31.987 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:28:31.988 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:28:31.988 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-02-05T05:28:31.988 INFO:tasks.workunit.client.0.smithi148.stderr:160667: _ 2024-02-05T05:28:31.988 INFO:tasks.workunit.client.0.smithi148.stderr:160667: hinfo_key 2024-02-05T05:28:31.988 INFO:tasks.workunit.client.0.smithi148.stderr:160667: snapset 2024-02-05T05:28:31.988 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T05:28:31.988 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:28:31.988 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:28:31.988 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:28:31.988 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:28:31.988 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:28:31.988 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:28:31.988 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:28:31.991 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:28:31.991 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:28:31.991 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:28:31.991 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:28:31.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:28:31.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:28:31.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:28:31.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:28:31.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:28:31.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:28:31.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:28:31.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:28:31.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:28:31.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:28:31.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:28:31.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:28:31.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:28:31.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:28:31.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:28:31.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:28:31.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:28:31.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:28:31.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:28:31.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:28:31.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:28:31.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:28:31.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: start osd.2 2024-02-05T05:28:32.014 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:28:32.014 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:28:32.014 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:28:32.014 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:28:32.014 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:28:32.014 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:28:32.014 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:28:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:28:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:28:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:28:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:28:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:28:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:28:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:28:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:28:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:28:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:28:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:28:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:28:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:28:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:28:32.016 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:28:32.016 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:28:32.016 INFO:tasks.workunit.client.0.smithi148.stderr:160662: _ 2024-02-05T05:28:32.016 INFO:tasks.workunit.client.0.smithi148.stderr:160662: hinfo_key 2024-02-05T05:28:32.016 INFO:tasks.workunit.client.0.smithi148.stderr:160662: snapset 2024-02-05T05:28:32.016 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:28:32.016 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:28:32.016 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:28:32.016 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:28:32.016 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:28:32.016 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:28:32.016 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:28:32.017 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:28:32.018 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:28:32.018 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:28:32.018 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:28:32.018 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:28:32.018 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:28:32.018 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:28:32.019 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:28:32.019 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:28:32.019 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:28:32.019 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:28:32.019 INFO:tasks.workunit.client.0.smithi148.stderr:160662: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:28:32.019 INFO:tasks.workunit.client.0.smithi148.stderr:160662: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:28:32.019 INFO:tasks.workunit.client.0.smithi148.stderr:160662: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:28:32.019 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:28:32.019 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:28:32.019 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:28:32.019 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:28:32.019 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:28:32.019 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:28:32.020 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:28:32.020 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:28:32.020 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:28:32.020 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:28:32.020 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:28:32.020 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:28:32.020 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:28:32.020 INFO:tasks.workunit.client.0.smithi148.stderr:160662: start osd.1 2024-02-05T05:28:36.806 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:28:36.806 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:28:36.806 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:28:36.807 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:28:36.807 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:28:36.807 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:28:36.807 INFO:tasks.workunit.client.0.smithi148.stderr:160662: 2024-02-05T05:28:32.048+0000 7f75ab42b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:36.807 INFO:tasks.workunit.client.0.smithi148.stderr:160662: 2024-02-05T05:28:32.052+0000 7f75ab42b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:36.807 INFO:tasks.workunit.client.0.smithi148.stderr:160662: 2024-02-05T05:28:32.052+0000 7f75ab42b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:36.807 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:28:36.807 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:28:36.807 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:28:36.807 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:28:36.807 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:28:36.807 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:36.807 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:28:36.808 INFO:tasks.workunit.client.0.smithi148.stderr:160662: 0 2024-02-05T05:28:36.808 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:36.808 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:36.808 INFO:tasks.workunit.client.0.smithi148.stderr:160662: 2024-02-05T05:28:32.592+0000 7f75ab42b640 -1 Falling back to public interface 2024-02-05T05:28:36.808 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:36.808 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:36.808 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:36.808 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:28:36.808 INFO:tasks.workunit.client.0.smithi148.stderr:160662: 1 2024-02-05T05:28:36.808 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:36.808 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:36.808 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:36.809 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:36.809 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:36.809 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:28:36.809 INFO:tasks.workunit.client.0.smithi148.stderr:160662: 2 2024-02-05T05:28:36.809 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:36.809 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:36.809 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:36.809 INFO:tasks.workunit.client.0.smithi148.stderr:160662: 2024-02-05T05:28:35.664+0000 7f75ab42b640 -1 osd.1 68 log_to_monitors true 2024-02-05T05:28:36.809 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:36.809 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:36.809 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:28:36.809 INFO:tasks.workunit.client.0.smithi148.stderr:160662: 3 2024-02-05T05:28:36.809 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:36.809 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:37.049 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/covery_ops 2024-02-05T05:28:37.049 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:28:37.049 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:28:37.049 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:28:37.049 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:28:37.049 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:28:37.049 INFO:tasks.workunit.client.0.smithi148.stderr:160667: 2024-02-05T05:28:32.028+0000 7fd505115640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:37.049 INFO:tasks.workunit.client.0.smithi148.stderr:160667: 2024-02-05T05:28:32.032+0000 7fd505115640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:37.049 INFO:tasks.workunit.client.0.smithi148.stderr:160667: 2024-02-05T05:28:32.036+0000 7fd505115640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:37.049 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:28:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:28:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:28:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:28:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:28:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:28:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:160667: 0 2024-02-05T05:28:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:28:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:160667: 2024-02-05T05:28:32.580+0000 7fd505115640 -1 Falling back to public interface 2024-02-05T05:28:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:28:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:160667: 1 2024-02-05T05:28:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:28:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:160667: 2024-02-05T05:28:34.672+0000 7fd505115640 -1 osd.2 68 log_to_monitors true 2024-02-05T05:28:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:28:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:160667: 2 2024-02-05T05:28:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:28:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:37.052 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:37.052 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:28:37.052 INFO:tasks.workunit.client.0.smithi148.stderr:160667: 3 2024-02-05T05:28:37.052 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:37.052 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:28:38.025 INFO:tasks.workunit.client.0.smithi148.stderr:160667: osd.2 up in weight 1 up_from 73 up_thru 73 down_at 69 last_clean_interval [66,68) [v2:127.0.0.1:6810/4052906301,v1:127.0.0.1:6811/4052906301] [v2:127.0.0.1:6812/4052906301,v1:127.0.0.1:6813/4052906301] exists,up f2ffc804-401e-4c2e-8f9e-8ee4bf3b4085 2024-02-05T05:28:38.025 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:28:38.025 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:28:38.025 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:28:38.026 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:28:38.026 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:28:38.026 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:28:38.026 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:28:38.026 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:28:38.026 INFO:tasks.workunit.client.0.smithi148.stderr:160667: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:28:38.026 INFO:tasks.workunit.client.0.smithi148.stderr:160667: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:28:38.026 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:28:38.026 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:28:38.026 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:28:38.026 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:28:38.026 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:28:38.026 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: 1 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: 2 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: 3' 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836538 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836538 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538' 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:38.027 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:28:38.028 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514498 2024-02-05T05:28:38.028 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514498 2024-02-05T05:28:38.028 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-326417514498' 2024-02-05T05:28:38.028 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:38.940 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_staeph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:38.940 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:38.940 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:38.940 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:28:38.940 INFO:tasks.workunit.client.0.smithi148.stderr:160662: 4 2024-02-05T05:28:38.940 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:38.940 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:38.941 INFO:tasks.workunit.client.0.smithi148.stderr:160662: osd.1 up in weight 1 up_from 76 up_thru 66 down_at 69 last_clean_interval [65,68) [v2:127.0.0.1:6818/1349352150,v1:127.0.0.1:6819/1349352150] [v2:127.0.0.1:6820/1349352150,v1:127.0.0.1:6821/1349352150] exists,up 4b7b8b56-0151-49dc-aa8b-391396fbe19b 2024-02-05T05:28:38.941 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:28:38.941 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:28:38.941 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:28:38.941 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:28:38.941 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:28:38.941 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:28:38.941 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:28:38.941 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:28:38.941 INFO:tasks.workunit.client.0.smithi148.stderr:160662: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:28:38.941 INFO:tasks.workunit.client.0.smithi148.stderr:160662: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:28:38.941 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:28:38.941 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:28:38.942 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:28:38.942 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:28:38.942 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:28:38.942 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:28:38.942 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:28:38.942 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:28:38.942 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:28:38.942 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:28:38.942 INFO:tasks.workunit.client.0.smithi148.stderr:160662: 1 2024-02-05T05:28:38.942 INFO:tasks.workunit.client.0.smithi148.stderr:160662: 2 2024-02-05T05:28:38.942 INFO:tasks.workunit.client.0.smithi148.stderr:160662: 3' 2024-02-05T05:28:38.942 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:28:38.942 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:38.943 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:28:38.943 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836539 2024-02-05T05:28:38.943 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836539 2024-02-05T05:28:38.943 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539' 2024-02-05T05:28:38.943 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:40.554 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalonts: ceph tell osd.2 flush_pg_stats 2024-02-05T05:28:40.555 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-02-05T05:28:40.555 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-02-05T05:28:40.555 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-326417514498 2-313532612610' 2024-02-05T05:28:40.555 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:40.555 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:28:40.555 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135884 2024-02-05T05:28:40.555 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135884 2024-02-05T05:28:40.555 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-326417514498 2-313532612610 3-244813135884' 2024-02-05T05:28:40.555 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:40.555 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836538 2024-02-05T05:28:40.555 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:40.556 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:28:40.556 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836538 2024-02-05T05:28:40.556 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:40.556 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836538 2024-02-05T05:28:40.556 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-02-05T05:28:40.556 INFO:tasks.workunit.client.0.smithi148.stderr:160667: waiting osd.0 seq 21474836538 2024-02-05T05:28:40.556 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:40.556 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836537 -lt 21474836538 2024-02-05T05:28:40.556 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:28:40.556 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:28:40.556 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:40.556 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836538 2024-02-05T05:28:40.556 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:40.556 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-326417514498 2024-02-05T05:28:40.557 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:40.557 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:28:40.557 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-326417514498 2024-02-05T05:28:40.557 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:40.557 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514498 2024-02-05T05:28:40.557 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 326417514498' 2024-02-05T05:28:40.557 INFO:tasks.workunit.client.0.smithi148.stderr:160667: waiting osd.1 seq 326417514498 2024-02-05T05:28:40.557 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:28:40.557 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514498 -lt 326417514498 2024-02-05T05:28:41.056 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in e/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:28:41.056 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514499 2024-02-05T05:28:41.056 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514499 2024-02-05T05:28:41.056 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-326417514499' 2024-02-05T05:28:41.056 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:41.056 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:28:41.057 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612611 2024-02-05T05:28:41.057 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612611 2024-02-05T05:28:41.057 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-326417514499 2-313532612611' 2024-02-05T05:28:41.057 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:41.057 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:28:41.057 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135885 2024-02-05T05:28:41.057 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135885 2024-02-05T05:28:41.057 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-326417514499 2-313532612611 3-244813135885' 2024-02-05T05:28:41.057 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:41.057 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836539 2024-02-05T05:28:41.057 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:41.057 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:28:41.058 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836539 2024-02-05T05:28:41.058 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:41.058 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836539 2024-02-05T05:28:41.058 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2024-02-05T05:28:41.058 INFO:tasks.workunit.client.0.smithi148.stderr:160662: waiting osd.0 seq 21474836539 2024-02-05T05:28:41.058 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:41.058 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836539 2024-02-05T05:28:41.058 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:28:41.058 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:28:41.058 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:41.058 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836539 2024-02-05T05:28:41.058 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:41.058 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-326417514499 2024-02-05T05:28:41.059 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:41.059 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:28:41.059 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-326417514499 2024-02-05T05:28:41.059 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:41.991 INFO:tasks.workunit.client.0.smithi148.stderr:16066$seqs 2024-02-05T05:28:41.991 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612610 2024-02-05T05:28:41.991 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:41.991 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:28:41.991 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612610 2024-02-05T05:28:41.991 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:41.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-02-05T05:28:41.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-02-05T05:28:41.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: waiting osd.2 seq 313532612610 2024-02-05T05:28:41.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:28:41.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612612 -lt 313532612610 2024-02-05T05:28:41.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:41.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-244813135884 2024-02-05T05:28:41.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:41.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:28:41.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-244813135884 2024-02-05T05:28:41.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:41.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135884 2024-02-05T05:28:41.992 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 244813135884' 2024-02-05T05:28:41.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: waiting osd.3 seq 244813135884 2024-02-05T05:28:41.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:28:41.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135885 -lt 244813135884 2024-02-05T05:28:41.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:28:41.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:41.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:41.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:28:41.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:28:41.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:28:41.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:28:41.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:28:41.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:28:41.993 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:28:41.994 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:28:41.994 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:28:41.994 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:28:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephte2: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514499 2024-02-05T05:28:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 326417514499' 2024-02-05T05:28:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:160662: waiting osd.1 seq 326417514499 2024-02-05T05:28:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:28:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514499 2024-02-05T05:28:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612611 2024-02-05T05:28:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:28:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612611 2024-02-05T05:28:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612611 2024-02-05T05:28:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612611' 2024-02-05T05:28:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:160662: waiting osd.2 seq 313532612611 2024-02-05T05:28:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:28:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612612 -lt 313532612611 2024-02-05T05:28:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-244813135885 2024-02-05T05:28:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:28:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-244813135885 2024-02-05T05:28:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135885 2024-02-05T05:28:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 244813135885' 2024-02-05T05:28:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:160662: waiting osd.3 seq 244813135885 2024-02-05T05:28:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:28:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135885 -lt 244813135885 2024-02-05T05:28:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:28:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:28:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:28:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:28:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:28:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:28:43.132 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:28:43.133 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:28:43.133 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:28:43.133 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:28:43.133 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:28:43.133 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:43.133 INFO:tasks.workunit.client.0.smithi148.stderr:160662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:43.133 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:28:43.133 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:28:43.133 INFO:tasks.workunit.client.0.smithi148.stderr:160662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:28:43.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:28:43.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:28:43.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 160663 2024-02-05T05:28:43.156 INFO:tasks.workunit.client.0.smithi148.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:43.156 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:43.156 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:28:43.156 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T05:28:43.156 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:28:43.156 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:28:43.156 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:28:43.156 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:28:43.156 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:28:43.156 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:28:43.157 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:28:43.157 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:28:43.157 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:28:43.157 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:28:43.157 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:28:43.157 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:28:43.157 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:28:43.157 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:28:43.157 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:43.157 INFO:tasks.workunit.client.0.smithi148.stderr:160667: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:43.157 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:28:43.157 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:28:43.157 INFO:tasks.workunit.client.0.smithi148.stderr:160667: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:28:43.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:28:43.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:28:43.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:28:43.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:28:43.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:28:43.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:28:43.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:28:43.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:28:43.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-02-05T05:28:43.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:28:43.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:28:43.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-02-05T05:28:43.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-02-05T05:28:43.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:28:43.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:28:43.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:28:43.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:28:43.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 163413"' 2024-02-05T05:28:43.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 163413' 2024-02-05T05:28:43.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:28:43.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:28:43.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:28:43.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 163415"' 2024-02-05T05:28:43.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 163415' 2024-02-05T05:28:43.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:28:43.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:28:43.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 163413 163415' 2024-02-05T05:28:43.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:28:43.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:28:43.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 163413 2024-02-05T05:28:43.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/163416: /' 2024-02-05T05:28:43.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:28:43.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/163418: /' 2024-02-05T05:28:46.483 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:28:46.483 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:28:46.483 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:28:46.483 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:28:46.483 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:28:46.483 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:28:46.483 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:28:46.483 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:28:46.483 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:28:46.484 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:28:46.484 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:28:46.484 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:28:46.484 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:28:46.484 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:28:46.484 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:28:46.484 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:28:46.484 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:28:46.484 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:28:46.484 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:28:46.484 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:28:46.484 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:28:46.484 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:28:46.484 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:28:46.485 INFO:tasks.workunit.client.0.smithi148.stderr:163418: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:28:46.485 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:28:46.485 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:28:46.485 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:28:46.485 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:28:46.485 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:28:46.485 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:28:46.485 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:28:46.485 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:28:46.488 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:28:46.488 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:28:46.489 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:28:46.489 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:28:46.489 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:28:46.489 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:28:46.489 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:28:46.489 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:28:46.489 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:28:46.489 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:28:46.489 INFO:tasks.workunit.client.0.smithi148.stderr:163418: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:28:46.489 INFO:tasks.workunit.client.0.smithi148.stderr:163418: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:28:46.489 INFO:tasks.workunit.client.0.smithi148.stderr:163418: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:28:46.490 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:28:46.490 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:28:46.490 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:28:46.490 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:28:46.490 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:28:46.490 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:28:46.490 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:28:46.490 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:28:46.490 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:28:46.491 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:28:46.491 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:28:46.491 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:28:46.491 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:28:46.491 INFO:tasks.workunit.client.0.smithi148.stderr:163418: start osd.1 2024-02-05T05:28:46.491 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:28:47.048 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:28:47.048 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:28:47.049 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:28:47.049 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:28:47.049 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:28:47.049 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:28:47.049 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:28:47.049 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:28:47.049 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:28:47.049 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:28:47.049 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:28:47.049 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:28:47.049 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:28:47.049 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:28:47.050 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:28:47.050 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:28:47.050 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:28:47.050 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:28:47.050 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:28:47.050 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:28:47.050 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:28:47.050 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:28:47.050 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:28:47.050 INFO:tasks.workunit.client.0.smithi148.stderr:163416: remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:28:47.051 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:28:47.051 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:28:47.051 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:28:47.051 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:28:47.051 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:28:47.051 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:28:47.051 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:28:47.051 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:28:47.052 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:28:47.052 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:28:47.052 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:28:47.052 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:28:47.052 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:28:47.052 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:28:47.053 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:28:47.053 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:28:47.053 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:28:47.053 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:28:47.053 INFO:tasks.workunit.client.0.smithi148.stderr:163416: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:28:47.053 INFO:tasks.workunit.client.0.smithi148.stderr:163416: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:28:47.053 INFO:tasks.workunit.client.0.smithi148.stderr:163416: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:28:47.053 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:28:47.053 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:28:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:28:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:28:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:28:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:28:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:28:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:28:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:28:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:28:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:28:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:28:47.054 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:28:47.055 INFO:tasks.workunit.client.0.smithi148.stderr:163416: start osd.3 2024-02-05T05:28:47.055 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:28:51.234 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:28:51.234 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:28:51.234 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:28:51.234 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:28:51.234 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:28:51.234 INFO:tasks.workunit.client.0.smithi148.stderr:163418: 2024-02-05T05:28:46.519+0000 7f12da490640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:51.234 INFO:tasks.workunit.client.0.smithi148.stderr:163418: 2024-02-05T05:28:46.523+0000 7f12da490640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:51.234 INFO:tasks.workunit.client.0.smithi148.stderr:163418: 2024-02-05T05:28:46.527+0000 7f12da490640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:51.234 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:28:51.234 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:28:51.235 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:28:51.235 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:28:51.235 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:28:51.235 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:51.235 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:28:51.235 INFO:tasks.workunit.client.0.smithi148.stderr:163418: 0 2024-02-05T05:28:51.235 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:51.235 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:51.235 INFO:tasks.workunit.client.0.smithi148.stderr:163418: 2024-02-05T05:28:47.067+0000 7f12da490640 -1 Falling back to public interface 2024-02-05T05:28:51.235 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:51.235 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:51.235 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:51.235 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:28:51.235 INFO:tasks.workunit.client.0.smithi148.stderr:163418: 1 2024-02-05T05:28:51.236 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:51.236 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:51.236 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:51.236 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:51.236 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:51.236 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:28:51.236 INFO:tasks.workunit.client.0.smithi148.stderr:163418: 2 2024-02-05T05:28:51.236 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:51.236 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:51.236 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:51.236 INFO:tasks.workunit.client.0.smithi148.stderr:163418: 2024-02-05T05:28:50.035+0000 7f12da490640 -1 osd.1 78 log_to_monitors true 2024-02-05T05:28:51.236 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:51.236 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:51.236 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:28:51.237 INFO:tasks.workunit.client.0.smithi148.stderr:163418: 3 2024-02-05T05:28:51.237 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:51.237 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:51.950 INFO:tasks.workunit.client.0.smithi148.stderr:163418: osd.1 up in weight 1 up_from 83 up_thru 76 down_at 79 last_163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:28:51.979 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:28:51.979 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:28:51.980 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:28:51.980 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:28:51.980 INFO:tasks.workunit.client.0.smithi148.stderr:163416: 2024-02-05T05:28:47.083+0000 7fdd018f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:51.980 INFO:tasks.workunit.client.0.smithi148.stderr:163416: 2024-02-05T05:28:47.087+0000 7fdd018f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:51.980 INFO:tasks.workunit.client.0.smithi148.stderr:163416: 2024-02-05T05:28:47.087+0000 7fdd018f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:51.980 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:28:51.980 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:28:51.980 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:28:51.980 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:28:51.980 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:28:51.980 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:51.980 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:28:51.981 INFO:tasks.workunit.client.0.smithi148.stderr:163416: 0 2024-02-05T05:28:51.981 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:51.981 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:28:51.981 INFO:tasks.workunit.client.0.smithi148.stderr:163416: 2024-02-05T05:28:47.619+0000 7fdd018f4640 -1 Falling back to public interface 2024-02-05T05:28:51.981 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:51.981 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:51.981 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:51.981 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:28:51.981 INFO:tasks.workunit.client.0.smithi148.stderr:163416: 1 2024-02-05T05:28:51.981 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:51.981 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:28:51.981 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:51.981 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:51.981 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:28:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:163416: 2 2024-02-05T05:28:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:28:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:163416: 2024-02-05T05:28:50.851+0000 7fdd018f4640 -1 osd.3 78 log_to_monitors true 2024-02-05T05:28:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:28:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:163416: 3 2024-02-05T05:28:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:28:52.367 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperclean_interval [76,78) [v2:127.0.0.1:6818/4292646121,v1:127.0.0.1:6819/4292646121] [v2:127.0.0.1:6820/4292646121,v1:127.0.0.1:6821/4292646121] exists,up 4b7b8b56-0151-49dc-aa8b-391396fbe19b 2024-02-05T05:28:52.367 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:28:52.367 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:28:52.367 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:28:52.367 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:28:52.367 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:28:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:28:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:28:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:28:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:163418: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:28:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:163418: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:28:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:28:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:28:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:28:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:28:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:28:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:28:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:28:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:28:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:28:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:28:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:163418: 1 2024-02-05T05:28:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:163418: 2 2024-02-05T05:28:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:163418: 3' 2024-02-05T05:28:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:28:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:28:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836543 2024-02-05T05:28:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836543 2024-02-05T05:28:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543' 2024-02-05T05:28:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:28:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-02-05T05:28:52.370 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-02-05T05:28:52.370 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-356482285570' 2024-02-05T05:28:52.370 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:53.925 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:28:53.925 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612615 2024-02-05T05:28:53.925 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612615 2024-02-05T05:28:53.925 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-356482285570 2-313532612615' 2024-02-05T05:28:53.925 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:53.925 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:28:53.925 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220162 2024-02-05T05:28:53.925 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220162 2024-02-05T05:28:53.925 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-356482285570 2-313532612615 3-365072220162' 2024-02-05T05:28:53.925 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:53.925 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836543 2024-02-05T05:28:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:28:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836543 2024-02-05T05:28:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836543 2024-02-05T05:28:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-02-05T05:28:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:163418: waiting osd.0 seq 21474836543 2024-02-05T05:28:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836543 2024-02-05T05:28:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-356482285570 2024-02-05T05:28:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:28:53.927 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-356482285570 2024-02-05T05:28:53.927 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:53.927 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-02-05T05:28:53.927 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 356482285570' 2024-02-05T05:28:53.927 INFO:tasks.workunit.client.0.smithi148.stderr:163418: waiting osd.1 seq 356482285570 2024-02-05T05:28:53.927 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:28:53.927 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285570 -lt 356482285570 2024-02-05T05:28:53.927 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:53.927 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612615 2024-02-05T05:28:53.927 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:53.927 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:28:53.927 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612615 2024-02-05T05:28:54.110 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephs.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:54.110 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:54.110 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:54.110 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:28:54.110 INFO:tasks.workunit.client.0.smithi148.stderr:163416: 4 2024-02-05T05:28:54.111 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:54.111 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:28:54.111 INFO:tasks.workunit.client.0.smithi148.stderr:163416: osd.3 up in weight 1 up_from 85 up_thru 85 down_at 79 last_clean_interval [57,78) [v2:127.0.0.1:6826/2874460624,v1:127.0.0.1:6827/2874460624] [v2:127.0.0.1:6828/2874460624,v1:127.0.0.1:6829/2874460624] exists,up c45cee7e-c47e-4e9f-a0cb-f1770a36dec6 2024-02-05T05:28:54.111 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:28:54.111 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:28:54.111 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:28:54.111 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:28:54.111 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:28:54.111 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:28:54.111 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:28:54.111 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:28:54.111 INFO:tasks.workunit.client.0.smithi148.stderr:163416: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:28:54.111 INFO:tasks.workunit.client.0.smithi148.stderr:163416: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:28:54.112 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:28:54.112 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:28:54.112 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:28:54.112 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:28:54.112 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:28:54.112 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:28:54.112 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:28:54.112 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:28:54.112 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:28:54.112 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:28:54.112 INFO:tasks.workunit.client.0.smithi148.stderr:163416: 1 2024-02-05T05:28:54.112 INFO:tasks.workunit.client.0.smithi148.stderr:163416: 2 2024-02-05T05:28:54.112 INFO:tasks.workunit.client.0.smithi148.stderr:163416: 3' 2024-02-05T05:28:54.113 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:28:54.113 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:54.113 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:28:54.113 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836544 2024-02-05T05:28:54.113 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836544 2024-02-05T05:28:54.113 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544' 2024-02-05T05:28:54.113 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:55.238 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:28:55.238 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285571 2024-02-05T05:28:55.238 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285571 2024-02-05T05:28:55.238 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-356482285571' 2024-02-05T05:28:55.238 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:55.238 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:28:55.238 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612616 2024-02-05T05:28:55.239 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612616 2024-02-05T05:28:55.239 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-356482285571 2-313532612616' 2024-02-05T05:28:55.239 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:55.239 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:28:55.239 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220163 2024-02-05T05:28:55.239 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220163 2024-02-05T05:28:55.239 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-356482285571 2-313532612616 3-365072220163' 2024-02-05T05:28:55.239 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:55.239 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836544 2024-02-05T05:28:55.239 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:55.239 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:28:55.239 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836544 2024-02-05T05:28:55.239 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:55.240 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836544 2024-02-05T05:28:55.240 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-02-05T05:28:55.240 INFO:tasks.workunit.client.0.smithi148.stderr:163416: waiting osd.0 seq 21474836544 2024-02-05T05:28:55.240 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:55.240 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836544 2024-02-05T05:28:55.240 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:55.240 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:55.240 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-356482285571 2024-02-05T05:28:55.240 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:28:55.240 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-356482285571 2024-02-05T05:28:55.240 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:55.240 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285571 2024-02-05T05:28:55.241 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 356482285571' 2024-02-05T05:28:55.241 INFO:tasks.workunit.client.0.smithi148.stderr:163416: waiting osd.1 seq 356482285571 2024-02-05T05:28:55.241 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:28:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285571 -lt 356482285571 2024-02-05T05:28:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612616 2024-02-05T05:28:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:28:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612616 2024-02-05T05:28:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612616 2024-02-05T05:28:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612616' 2024-02-05T05:28:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:163416: waiting osd.2 seq 313532612616 2024-02-05T05:28:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:28:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612616 -lt 313532612616 2024-02-05T05:28:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-365072220163 2024-02-05T05:28:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:28:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-365072220163 2024-02-05T05:28:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220163 2024-02-05T05:28:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 365072220163' 2024-02-05T05:28:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:163416: waiting osd.3 seq 365072220163 2024-02-05T05:28:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:28:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220163 2024-02-05T05:28:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:28:56.604 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:56.604 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:56.604 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:28:56.604 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:28:56.604 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:28:56.604 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:28:56.604 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:28:56.604 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:28:56.604 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:28:56.604 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:28:56.605 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_cleatest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:56.605 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612615 2024-02-05T05:28:56.605 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612615' 2024-02-05T05:28:56.605 INFO:tasks.workunit.client.0.smithi148.stderr:163418: waiting osd.2 seq 313532612615 2024-02-05T05:28:56.605 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:28:56.605 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612615 -lt 313532612615 2024-02-05T05:28:56.605 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:56.605 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-365072220162 2024-02-05T05:28:56.605 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:56.605 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:28:56.605 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-365072220162 2024-02-05T05:28:56.605 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220162 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 365072220162' 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: waiting osd.3 seq 365072220162 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 365072220162 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220162 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:28:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:28:56.607 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:28:56.607 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:28:56.607 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:28:56.607 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:28:56.607 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:28:56.607 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:28:56.607 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:28:56.966 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:56.966 INFO:tasks.workunit.client.0.smithi148.stderr:163418: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:56.966 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:28:56.967 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:28:56.967 INFO:tasks.workunit.client.0.smithi148.stderr:163418: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:28:56.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:28:56.982 INFO:tasks.workunit.client.0.smithi148.stderr:n: cur_active_clean=5 2024-02-05T05:28:56.982 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:28:56.982 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:56.983 INFO:tasks.workunit.client.0.smithi148.stderr:163416: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:56.983 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:28:56.983 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:28:56.983 INFO:tasks.workunit.client.0.smithi148.stderr:163416: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:28:56.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:28:56.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:28:56.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 163415 2024-02-05T05:28:56.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:28:56.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:28:56.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:28:56.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:28:56.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:28:56.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:28:56.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:28:56.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:28:56.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:28:56.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:28:57.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:28:57.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:28:57.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:28:57.310 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:28:57.310 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:28:57.310 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:28:57.310 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:28:57.310 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:28:57.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:28:25.579903+0000 2024-02-05T05:28:57.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:28:57.914 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:28:57.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:28:25.579903+0000 2024-02-05T05:28:57.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:28:57.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:28:25.579903+0000 2024-02-05T05:28:57.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:28:57.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:28:57.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:28:57.931 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:28:57.931 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:28:57.931 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:28:57.931 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:28:57.931 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:28:58.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:25.579903+0000 '>' 2024-02-05T05:28:25.579903+0000 2024-02-05T05:28:58.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:28:59.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:28:59.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:28:59.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:28:59.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:28:59.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:28:59.221 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:28:59.221 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:28:59.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:25.579903+0000 '>' 2024-02-05T05:28:25.579903+0000 2024-02-05T05:28:59.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:29:00.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:29:00.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:29:00.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:29:00.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:29:00.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:29:00.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:29:00.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:29:00.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:25.579903+0000 '>' 2024-02-05T05:28:25.579903+0000 2024-02-05T05:29:00.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:29:01.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:29:01.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:29:01.859 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:29:01.859 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:29:01.859 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:29:01.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:29:01.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:29:02.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:25.579903+0000 '>' 2024-02-05T05:28:25.579903+0000 2024-02-05T05:29:02.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:29:03.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:29:03.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:29:03.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:29:03.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:29:03.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:29:03.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:29:03.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:29:03.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:58.743693+0000 '>' 2024-02-05T05:28:25.579903+0000 2024-02-05T05:29:03.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:29:03.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:29:03.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:29:03.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:29:03.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:29:03.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 166267"' 2024-02-05T05:29:03.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 166267' 2024-02-05T05:29:03.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:29:03.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:29:03.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:29:03.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 166269"' 2024-02-05T05:29:03.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 166269' 2024-02-05T05:29:03.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:29:03.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 166267 166269' 2024-02-05T05:29:03.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:29:03.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:29:03.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/166270: /' 2024-02-05T05:29:03.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 166267 2024-02-05T05:29:03.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:29:03.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:29:03.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/166272: /' 2024-02-05T05:29:06.514 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:29:06.515 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:29:06.515 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:29:06.515 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:29:06.515 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:29:06.515 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:29:06.515 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:29:06.515 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:29:06.515 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:29:06.515 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:29:06.515 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:29:06.515 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:29:06.516 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:29:06.516 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:29:06.516 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:29:06.516 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:29:06.516 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:29:06.516 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:29:06.516 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:29:06.516 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:29:06.516 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:29:06.516 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:29:06.516 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:29:06.517 INFO:tasks.workunit.client.0.smithi148.stderr:166270: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:29:06.517 INFO:tasks.workunit.client.0.smithi148.stderr:166270: _ 2024-02-05T05:29:06.517 INFO:tasks.workunit.client.0.smithi148.stderr:166270: hinfo_key 2024-02-05T05:29:06.517 INFO:tasks.workunit.client.0.smithi148.stderr:166270: snapset 2024-02-05T05:29:06.517 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:29:06.517 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:29:06.517 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:29:06.517 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:29:06.517 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:29:06.517 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:29:06.517 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:29:06.520 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:29:06.520 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:29:06.520 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:29:06.520 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:29:06.520 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:29:06.520 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:29:06.520 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:29:06.520 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:29:06.520 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:29:06.521 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:29:06.522 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:29:06.522 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:29:06.522 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:29:06.522 INFO:tasks.workunit.client.0.smithi148.stderr:166270: start osd.3 2024-02-05T05:29:07.086 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:29:07.086 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:29:07.086 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:29:07.086 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:29:07.086 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:29:07.086 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:29:07.086 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:29:07.087 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:29:07.087 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:29:07.087 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:29:07.087 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:29:07.087 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:29:07.087 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:29:07.087 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:29:07.087 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:29:07.087 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:29:07.087 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:29:07.087 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:29:07.087 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:29:07.087 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:29:07.088 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:29:07.088 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:29:07.088 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:29:07.088 INFO:tasks.workunit.client.0.smithi148.stderr:166272: _ 2024-02-05T05:29:07.088 INFO:tasks.workunit.client.0.smithi148.stderr:166272: hinfo_key 2024-02-05T05:29:07.088 INFO:tasks.workunit.client.0.smithi148.stderr:166272: snapset 2024-02-05T05:29:07.088 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:29:07.088 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:29:07.088 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:29:07.088 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:29:07.088 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:29:07.088 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:29:07.088 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:29:07.088 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:29:07.090 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:29:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:29:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:29:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:29:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:29:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:29:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:29:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:29:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:29:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:29:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:166272: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:29:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:166272: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:166272: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:29:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:29:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:29:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:29:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:29:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:29:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:29:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:29:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:29:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:29:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:29:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:29:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:29:07.093 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:29:07.093 INFO:tasks.workunit.client.0.smithi148.stderr:166272: start osd.1 2024-02-05T05:29:10.774 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:29:10.774 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:29:10.774 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:29:10.774 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:29:10.774 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:29:10.774 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:29:10.774 INFO:tasks.workunit.client.0.smithi148.stderr:166270: 2024-02-05T05:29:06.551+0000 7fe63cea7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:10.775 INFO:tasks.workunit.client.0.smithi148.stderr:166270: 2024-02-05T05:29:06.555+0000 7fe63cea7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:10.775 INFO:tasks.workunit.client.0.smithi148.stderr:166270: 2024-02-05T05:29:06.555+0000 7fe63cea7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:10.775 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:29:10.775 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:29:10.775 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:29:10.775 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:29:10.775 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:29:10.775 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:10.775 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:29:10.775 INFO:tasks.workunit.client.0.smithi148.stderr:166270: 0 2024-02-05T05:29:10.775 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:10.775 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:29:10.775 INFO:tasks.workunit.client.0.smithi148.stderr:166270: 2024-02-05T05:29:07.103+0000 7fe63cea7640 -1 Falling back to public interface 2024-02-05T05:29:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:29:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:166270: 1 2024-02-05T05:29:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:29:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:29:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:166270: 2 2024-02-05T05:29:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:10.777 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:29:10.777 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:10.777 INFO:tasks.workunit.client.0.smithi148.stderr:166270: 2024-02-05T05:29:09.955+0000 7fe63cea7640 -1 osd.3 87 log_to_monitors true 2024-02-05T05:29:10.777 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:10.777 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:10.777 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:29:10.777 INFO:tasks.workunit.client.0.smithi148.stderr:166270: 3 2024-02-05T05:29:10.777 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:12.001 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-02-05T05:29:12.002 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:29:12.002 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:29:12.002 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:29:12.002 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:29:12.002 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:29:12.002 INFO:tasks.workunit.client.0.smithi148.stderr:166272: 2024-02-05T05:29:07.119+0000 7fcc81c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:12.002 INFO:tasks.workunit.client.0.smithi148.stderr:166272: 2024-02-05T05:29:07.123+0000 7fcc81c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:12.002 INFO:tasks.workunit.client.0.smithi148.stderr:166272: 2024-02-05T05:29:07.123+0000 7fcc81c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:12.002 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:29:12.002 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:29:12.002 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:29:12.002 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:29:12.002 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: 0 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: 2024-02-05T05:29:07.675+0000 7fcc81c17640 -1 Falling back to public interface 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: 1 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:12.003 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:29:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:166272: 2 2024-02-05T05:29:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:166272: 2024-02-05T05:29:10.555+0000 7fcc81c17640 -1 osd.1 87 log_to_monitors true 2024-02-05T05:29:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:29:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:166272: 3 2024-02-05T05:29:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:12.311 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:29:12.311 INFO:tasks.workunit.client.0.smithi148.stderr:166270: osd.3 up in weight 1 up_from 92 up_thru 85 down_at 88 last_clean_interval [85,87) [v2:127.0.0.1:6818/4047586187,v1:127.0.0.1:6819/4047586187] [v2:127.0.0.1:6820/4047586187,v1:127.0.0.1:6821/4047586187] exists,up c45cee7e-c47e-4e9f-a0cb-f1770a36dec6 2024-02-05T05:29:12.311 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:29:12.311 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:29:12.311 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:29:12.311 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:29:12.311 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:29:12.312 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:29:12.312 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:29:12.312 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:29:12.312 INFO:tasks.workunit.client.0.smithi148.stderr:166270: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:29:12.312 INFO:tasks.workunit.client.0.smithi148.stderr:166270: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:29:12.312 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:29:12.312 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:29:12.312 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:29:12.312 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:29:12.312 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:29:12.312 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:29:12.312 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:29:12.312 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: 1 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: 2 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: 3' 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836549 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836549 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549' 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-02-05T05:29:12.313 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-02-05T05:29:12.314 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-408021893122' 2024-02-05T05:29:13.821 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:29:13.822 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:29:13.822 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612621 2024-02-05T05:29:13.822 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612621 2024-02-05T05:29:13.822 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-408021893122 2-313532612621' 2024-02-05T05:29:13.822 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:29:13.822 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:29:13.822 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991234 2024-02-05T05:29:13.822 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991234 2024-02-05T05:29:13.822 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-408021893122 2-313532612621 3-395136991234' 2024-02-05T05:29:13.822 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:29:13.822 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836549 2024-02-05T05:29:13.822 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:29:13.822 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:29:13.823 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836549 2024-02-05T05:29:13.823 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:29:13.823 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836549 2024-02-05T05:29:13.823 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836549' 2024-02-05T05:29:13.823 INFO:tasks.workunit.client.0.smithi148.stderr:166270: waiting osd.0 seq 21474836549 2024-02-05T05:29:13.823 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:29:13.823 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836549 2024-02-05T05:29:13.823 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:29:13.823 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-408021893122 2024-02-05T05:29:13.823 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:29:13.823 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:29:13.823 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-408021893122 2024-02-05T05:29:13.823 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:29:13.823 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-02-05T05:29:13.824 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 408021893122' 2024-02-05T05:29:13.824 INFO:tasks.workunit.client.0.smithi148.stderr:166270: waiting osd.1 seq 408021893122 2024-02-05T05:29:13.824 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:29:13.824 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893122 -lt 408021893122 2024-02-05T05:29:13.824 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:29:13.824 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:29:13.824 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612621 2024-02-05T05:29:13.824 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:29:14.070 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/cloeph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:14.070 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:14.070 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:14.070 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:29:14.070 INFO:tasks.workunit.client.0.smithi148.stderr:166272: 4 2024-02-05T05:29:14.070 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:14.070 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:14.070 INFO:tasks.workunit.client.0.smithi148.stderr:166272: osd.1 up in weight 1 up_from 95 up_thru 95 down_at 88 last_clean_interval [83,87) [v2:127.0.0.1:6826/1146153046,v1:127.0.0.1:6827/1146153046] [v2:127.0.0.1:6828/1146153046,v1:127.0.0.1:6829/1146153046] exists,up 4b7b8b56-0151-49dc-aa8b-391396fbe19b 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:29:14.071 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:166272: 1 2024-02-05T05:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:166272: 2 2024-02-05T05:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:166272: 3' 2024-02-05T05:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836550 2024-02-05T05:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836550 2024-02-05T05:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550' 2024-02-05T05:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:29:15.247 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:29:15.247 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893123 2024-02-05T05:29:15.247 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893123 2024-02-05T05:29:15.247 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-408021893123' 2024-02-05T05:29:15.247 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:29:15.247 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:29:15.247 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612622 2024-02-05T05:29:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612622 2024-02-05T05:29:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-408021893123 2-313532612622' 2024-02-05T05:29:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:29:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:29:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991235 2024-02-05T05:29:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991235 2024-02-05T05:29:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-408021893123 2-313532612622 3-395136991235' 2024-02-05T05:29:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:29:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836550 2024-02-05T05:29:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:29:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:29:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836550 2024-02-05T05:29:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:29:15.249 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836550 2024-02-05T05:29:15.249 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-02-05T05:29:15.249 INFO:tasks.workunit.client.0.smithi148.stderr:166272: waiting osd.0 seq 21474836550 2024-02-05T05:29:15.249 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:29:15.249 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836551 -lt 21474836550 2024-02-05T05:29:15.249 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:29:15.249 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-408021893123 2024-02-05T05:29:15.249 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:29:15.249 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:29:15.249 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-408021893123 2024-02-05T05:29:15.249 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:29:15.249 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893123 2024-02-05T05:29:15.249 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 408021893123' 2024-02-05T05:29:15.250 INFO:tasks.workunit.client.0.smithi148.stderr:166272: waiting osd.1 seq 408021893123 2024-02-05T05:29:15.250 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:29:16.806 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpne.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612621 2024-02-05T05:29:16.806 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:29:16.806 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612621 2024-02-05T05:29:16.806 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612621' 2024-02-05T05:29:16.806 INFO:tasks.workunit.client.0.smithi148.stderr:166270: waiting osd.2 seq 313532612621 2024-02-05T05:29:16.806 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:29:16.806 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612621 -lt 313532612621 2024-02-05T05:29:16.806 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991234 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991234 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991234 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991234' 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: waiting osd.3 seq 395136991234 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 395136991234 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991235 -lt 395136991234 2024-02-05T05:29:16.807 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:29:16.808 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:29:16.808 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:29:16.808 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:29:16.808 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:29:16.808 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:29:16.808 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:29:16.808 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:29:16.808 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:29:16.808 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:29:16.808 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:29:16.808 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:29:16.809 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qers.sh:2214: flush_pg_stats: test 408021893123 -lt 408021893123 2024-02-05T05:29:16.809 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:29:16.809 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612622 2024-02-05T05:29:16.809 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:29:16.809 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:29:16.809 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612622 2024-02-05T05:29:16.809 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:29:16.809 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612622 2024-02-05T05:29:16.809 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612622' 2024-02-05T05:29:16.809 INFO:tasks.workunit.client.0.smithi148.stderr:166272: waiting osd.2 seq 313532612622 2024-02-05T05:29:16.809 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:29:16.809 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612622 -lt 313532612622 2024-02-05T05:29:16.810 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:29:16.810 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:29:16.810 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991235 2024-02-05T05:29:16.810 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:29:16.810 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991235 2024-02-05T05:29:16.810 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:29:16.810 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991235 2024-02-05T05:29:16.810 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991235' 2024-02-05T05:29:16.810 INFO:tasks.workunit.client.0.smithi148.stderr:166272: waiting osd.3 seq 395136991235 2024-02-05T05:29:16.810 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:29:16.810 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991235 -lt 395136991235 2024-02-05T05:29:16.810 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:29:16.810 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:29:16.811 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:29:16.811 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:29:16.811 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:29:16.811 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:29:16.811 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:29:16.811 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:29:16.811 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:29:16.811 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:29:16.811 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:29:17.159 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:29:17.159 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:29:17.159 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:29:17.159 INFO:tasks.workunit.client.0.smithi148.stderr:166272: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:29:17.159 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:29:17.159 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:29:17.159 INFO:tasks.workunit.client.0.smithi148.stderr:166272: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:29:17.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:29:17.181 INFO:tasks.workunit.client.0.smithi148.stderr:a/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:29:17.181 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:29:17.181 INFO:tasks.workunit.client.0.smithi148.stderr:166270: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:29:17.181 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:29:17.181 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:29:17.181 INFO:tasks.workunit.client.0.smithi148.stderr:166270: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:29:17.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:29:17.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:29:17.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 166269 2024-02-05T05:29:17.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:29:17.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:29:17.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:29:17.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:29:17.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:29:17.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:29:17.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:29:17.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:29:17.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:29:17.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:29:17.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:29:17.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:29:17.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:29:17.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:29:17.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:29:17.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:29:17.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:29:17.573 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:29:17.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:29:17.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:29:17.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:29:17.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:29:17.578 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:29:17.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:29:17.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:29:17.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:29:17.581 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:29:17.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:29:17.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:29:17.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:29:17.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:29:17.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:17.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:29:17.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:29:17.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:29:17.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:29:17.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:29:17.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:29:17.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:29:17.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:29:17.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:29:17.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:29:17.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:29:17.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:29:17.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:29:17.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:29:17.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:29:17.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:29:17.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:29:17.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:29:17.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:29:17.626 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:29:17.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:29:17.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:29:17.627 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:29:17.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:29:17.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:29:17.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:29:17.629 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:29:17.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:29:17.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:29:17.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:29:17.633 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:29:17.633 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:17.633 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:29:17.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:29:17.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:29:17.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:29:17.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:29:17.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:29:17.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:17.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:29:17.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T05:29:17.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:29:17.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:29:17.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:29:17.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:29:17.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-02-05T05:29:17.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-02-05T05:29:17.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-02-05T05:29:17.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-02-05T05:29:17.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-02-05T05:29:17.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-02-05T05:29:17.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-02-05T05:29:17.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:29:17.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:29:17.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:29:17.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:29:17.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:29:17.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:29:18.055 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:29:18.055 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:29:18.055 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:29:18.056 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:29:18.056 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:18.056 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:29:18.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:29:18.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:29:18.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:29:18.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:29:18.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:29:18.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:29:18.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:29:18.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:29:18.278 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:29:18.279 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:29:18.279 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:29:18.279 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:29:18.279 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:18.279 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:29:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:29:18.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:29:18.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T05:29:18.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:29:18.398 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:29:18.398 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:29:18.398 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:29:18.398 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:29:18.398 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:29:18.398 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:29:18.398 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:29:18.399 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:29:18.399 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:18.399 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:29:18.399 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:29:18.400 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:29:18.400 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T05:29:18.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-02-05T05:29:18.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:29:18.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:29:18.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:29:18.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:29:18.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:29:18.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:29:19.392 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:29:19.392 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:29:19.392 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:29:19.393 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:29:19.393 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:19.393 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:29:19.393 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:29:19.393 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:29:19.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:29:19.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-02-05T05:29:19.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:29:19.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-02-05T05:29:19.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:29:19.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:29:19.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:29:19.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:29:19.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:29:19.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:29:19.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:29:19.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:29:19.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:29:19.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:29:19.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:29:19.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:29:19.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:29:19.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:29:19.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:29:19.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:29:19.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:29:19.437 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:29:19.437 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:19.437 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:29:19.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:29:19.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:29:19.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:29:19.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:29:19.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:29:19.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:29:19.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:29:19.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:29:19.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:29:19.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:29:19.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:29:19.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:29:19.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:29:19.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:29:19.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:29:19.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b2f7d09c-8d68-4b97-973b-0a6bb026c4ce 2024-02-05T05:29:19.439 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 b2f7d09c-8d68-4b97-973b-0a6bb026c4ce 2024-02-05T05:29:19.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b2f7d09c-8d68-4b97-973b-0a6bb026c4ce' 2024-02-05T05:29:19.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:29:19.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAvcsBlmHc8GxAAMjveYhgk88GNRwNPGndn+g== 2024-02-05T05:29:19.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAvcsBlmHc8GxAAMjveYhgk88GNRwNPGndn+g=="}' 2024-02-05T05:29:19.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b2f7d09c-8d68-4b97-973b-0a6bb026c4ce -i td/osd-scrub-repair/0/new.json 2024-02-05T05:29:19.720 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:29:19.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:29:19.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvcsBlmHc8GxAAMjveYhgk88GNRwNPGndn+g== --osd-uuid b2f7d09c-8d68-4b97-973b-0a6bb026c4ce 2024-02-05T05:29:19.772 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:19.770+0000 7f44ae07a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:19.788 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:19.786+0000 7f44ae07a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:19.789 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:19.786+0000 7f44ae07a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:19.790 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:19.786+0000 7f44ae07a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:29:19.790 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:19.786+0000 7f44ae07a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:29:19.790 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:19.786+0000 7f44ae07a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:29:19.888 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:19.886+0000 7f44ae07a640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:29:26.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:29:26.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:29:26.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:29:26.183 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:29:26.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:29:26.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:29:26.556 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:29:26.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:29:26.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:29:26.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:29:26.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:29:26.583 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:26.578+0000 7fa8cc3cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:26.585 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:26.582+0000 7fa8cc3cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:26.586 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:26.582+0000 7fa8cc3cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:26.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:29:26.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:29:26.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:29:26.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:29:26.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:29:26.878 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:29:26.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:26.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:29:26.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:26.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:27.130 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:27.126+0000 7fa8cc3cb640 -1 Falling back to public interface 2024-02-05T05:29:27.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:28.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:28.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:28.240 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:29:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:29:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:28.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:28.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:29.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:29.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:29.553 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:29:29.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:29:29.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:29.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:29.784 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:29.782+0000 7fa8cc3cb640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:29:29.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:30.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:30.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:30.905 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:29:30.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:29:30.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:30.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:31.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:32.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:32.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:32.223 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:29:32.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:29:32.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:32.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:32.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:33.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:33.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:33.690 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:29:33.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:29:33.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:33.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:34.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:35.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:35.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:35.008 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:29:35.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:29:35.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:35.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:35.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:36.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:36.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:36.318 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:29:36.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:29:36.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:36.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:36.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:37.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:37.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:37.690 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:29:37.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:29:37.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:37.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:38.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:39.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:39.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:39.017 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:29:39.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:29:39.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:39.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:39.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:40.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:40.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:40.337 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:29:40.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:29:40.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:40.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:40.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:41.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:41.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:41.680 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:29:41.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:29:41.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:41.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:41.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:42.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:42.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:42.996 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:29:42.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:29:42.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:42.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:43.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:44.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:44.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:44.310 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:29:44.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:29:44.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:44.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:44.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:45.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:45.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:45.651 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:29:45.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:29:45.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:45.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:46.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:47.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:47.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:47.062 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:29:47.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:29:47.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:47.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:47.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:48.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:48.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:48.547 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:29:48.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:29:48.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:48.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:48.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:49.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:49.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:49.954 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:29:49.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:29:49.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:49.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:50.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:51.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:51.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:51.916 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:29:51.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:29:51.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:51.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:52.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:53.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:53.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:53.227 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:29:53.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:29:53.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:53.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:53.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:54.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:54.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:54.942 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:29:54.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:29:54.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:54.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:55.325 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4220326303,v1:127.0.0.1:6803/4220326303] [v2:127.0.0.1:6804/4220326303,v1:127.0.0.1:6805/4220326303] exists,up b2f7d09c-8d68-4b97-973b-0a6bb026c4ce 2024-02-05T05:29:55.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:29:55.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:29:55.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:29:55.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:29:55.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-02-05T05:29:55.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:29:55.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:29:55.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:29:55.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:29:55.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:29:55.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:29:55.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:29:55.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:29:55.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:29:55.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:29:55.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:29:55.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:29:55.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:29:55.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:29:55.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:29:55.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:29:55.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:29:55.330 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:29:55.330 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:55.330 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:29:55.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:29:55.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:29:55.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:29:55.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:29:55.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:29:55.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:29:55.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:29:55.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:29:55.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:29:55.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:29:55.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:29:55.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:29:55.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:29:55.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:29:55.332 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:29:55.332 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 adc22c43-3a38-4d23-933d-b1fb1e5ffc07 2024-02-05T05:29:55.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=adc22c43-3a38-4d23-933d-b1fb1e5ffc07 2024-02-05T05:29:55.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 adc22c43-3a38-4d23-933d-b1fb1e5ffc07' 2024-02-05T05:29:55.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:29:55.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTcsBlSfHmFBAAP7uqDu541DEGQkVZ1Ms/gg== 2024-02-05T05:29:55.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTcsBlSfHmFBAAP7uqDu541DEGQkVZ1Ms/gg=="}' 2024-02-05T05:29:55.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new adc22c43-3a38-4d23-933d-b1fb1e5ffc07 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:29:55.715 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:29:55.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:29:55.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTcsBlSfHmFBAAP7uqDu541DEGQkVZ1Ms/gg== --osd-uuid adc22c43-3a38-4d23-933d-b1fb1e5ffc07 2024-02-05T05:29:55.763 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:55.761+0000 7f97f8b8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:55.818 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:55.817+0000 7f97f8b8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:55.819 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:55.817+0000 7f97f8b8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:55.820 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:55.821+0000 7f97f8b8c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:29:55.820 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:55.821+0000 7f97f8b8c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:29:55.820 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:55.821+0000 7f97f8b8c640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:29:55.896 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:29:55.897+0000 7f97f8b8c640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:30:02.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:30:02.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:30:02.679 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:30:02.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:30:02.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:30:03.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:30:03.068 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:30:03.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:30:03.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:30:03.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:30:03.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:30:03.099 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:03.097+0000 7f6cbf517640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:03.102 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:03.101+0000 7f6cbf517640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:03.103 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:03.101+0000 7f6cbf517640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:03.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:30:03.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:30:03.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:30:03.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:30:03.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:30:03.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:03.407 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:30:03.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:30:03.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:03.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:03.655 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:03.653+0000 7f6cbf517640 -1 Falling back to public interface 2024-02-05T05:30:03.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:04.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:04.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:04.753 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:30:04.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:30:04.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:04.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:05.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:06.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:06.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:06.053 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:30:06.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:30:06.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:06.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:06.342 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:06.341+0000 7f6cbf517640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:30:06.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:07.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:07.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:07.380 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:30:07.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:30:07.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:07.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:07.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:08.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:08.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:08.693 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:30:08.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:30:08.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:08.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:09.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:10.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:10.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:10.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:30:10.082 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:30:10.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:10.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:10.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:11.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:11.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:11.401 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:30:11.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:30:11.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:11.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:11.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:12.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:12.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:12.722 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:30:12.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:30:12.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:13.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:14.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:14.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:14.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:30:14.141 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:30:14.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:14.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:14.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:15.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:15.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:15.614 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:30:15.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:30:15.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:15.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:15.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:16.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:16.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:16.968 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:30:16.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:30:16.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:16.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:17.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:18.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:18.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:18.303 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:30:18.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:30:18.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:18.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:18.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:19.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:19.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:19.618 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:30:19.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:30:19.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:19.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:19.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:20.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:20.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:20.990 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:30:20.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:30:20.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:20.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:21.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:22.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:22.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:22.391 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:30:22.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:30:22.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:22.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:23.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:24.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:24.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:24.053 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:30:24.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:30:24.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:24.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:24.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:25.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:25.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:25.485 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:30:25.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:30:25.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:25.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:25.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:26.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:26.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:26.986 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:30:26.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:30:26.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:26.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:27.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:28.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:28.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:28.727 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:30:28.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:30:28.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:28.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:29.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:30.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:30.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:30.198 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:30:30.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:30:30.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:30.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:30.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:31.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:31.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:31.952 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:30:31.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:30:31.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:31.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:32.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:33.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:33.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:33.426 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T05:30:33.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T05:30:33.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:33.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:33.757 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3156156882,v1:127.0.0.1:6811/3156156882] [v2:127.0.0.1:6812/3156156882,v1:127.0.0.1:6813/3156156882] exists,up adc22c43-3a38-4d23-933d-b1fb1e5ffc07 2024-02-05T05:30:33.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:30:33.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:30:33.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:30:33.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:30:33.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-02-05T05:30:33.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:30:33.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:30:33.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:30:33.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:30:33.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:30:33.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:30:33.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:30:33.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:30:33.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:30:33.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:30:33.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:30:33.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:30:33.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:30:33.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:30:33.762 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:30:33.762 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:30:33.762 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:30:33.762 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:30:33.762 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:30:33.762 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:30:33.762 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:30:33.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:30:33.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:30:33.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:30:33.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:30:33.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:30:33.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:30:33.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:30:33.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:30:33.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:30:33.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:30:33.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:30:33.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:30:33.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:30:33.764 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:30:33.764 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 e636a31a-b223-4902-beab-826a6cd4b5d7 2024-02-05T05:30:33.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e636a31a-b223-4902-beab-826a6cd4b5d7 2024-02-05T05:30:33.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e636a31a-b223-4902-beab-826a6cd4b5d7' 2024-02-05T05:30:33.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:30:33.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5csBlQpXGLhAApJX5oB4aiHp8ZwRP/wGdQw== 2024-02-05T05:30:33.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5csBlQpXGLhAApJX5oB4aiHp8ZwRP/wGdQw=="}' 2024-02-05T05:30:33.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e636a31a-b223-4902-beab-826a6cd4b5d7 -i td/osd-scrub-repair/2/new.json 2024-02-05T05:30:34.151 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:30:34.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:30:34.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5csBlQpXGLhAApJX5oB4aiHp8ZwRP/wGdQw== --osd-uuid e636a31a-b223-4902-beab-826a6cd4b5d7 2024-02-05T05:30:34.205 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:34.204+0000 7efeae09b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:34.254 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:34.252+0000 7efeae09b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:34.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:34.256+0000 7efeae09b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:34.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:34.256+0000 7efeae09b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:30:34.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:34.256+0000 7efeae09b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:30:34.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:34.256+0000 7efeae09b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:30:34.337 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:34.336+0000 7efeae09b640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:30:40.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:30:40.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:30:41.011 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T05:30:41.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:30:41.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:30:41.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:30:41.365 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T05:30:41.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:30:41.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:30:41.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:30:41.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:30:41.399 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:41.396+0000 7f7912286640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:41.435 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:41.432+0000 7f7912286640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:41.437 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:41.436+0000 7f7912286640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:41.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:30:41.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:30:41.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:30:41.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:30:41.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:30:41.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:41.700 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:30:41.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:30:41.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:41.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:42.007 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:42.004+0000 7f7912286640 -1 Falling back to public interface 2024-02-05T05:30:42.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:43.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:43.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:43.026 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:30:43.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:30:43.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:43.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:43.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:44.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:44.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:44.341 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:30:44.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:30:44.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:44.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:44.487 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:30:44.484+0000 7f7912286640 -1 osd.2 0 log_to_monitors true 2024-02-05T05:30:44.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:45.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:45.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:45.723 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:30:45.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:30:45.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:45.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:46.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:47.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:47.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:47.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:30:47.041 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:30:47.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:47.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:47.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:48.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:48.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:48.435 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:30:48.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:30:48.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:48.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:48.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:49.776 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:30:49.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:30:49.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:49.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:50.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:51.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:51.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:51.154 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:30:51.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:30:51.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:51.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:51.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:52.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:52.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:52.600 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:30:52.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:30:52.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:52.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:52.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:53.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:53.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:53.997 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:30:53.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:30:53.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:53.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:54.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:55.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:55.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:55.459 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:30:55.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:30:55.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:55.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:55.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:56.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:56.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:56.771 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:30:56.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:30:56.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:56.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:57.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:58.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:58.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:58.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:30:58.212 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:30:58.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:58.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:58.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:59.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:59.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:59.756 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:30:59.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:30:59.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:59.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:00.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:01.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:01.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:01.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:31:01.222 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:31:01.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:01.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:01.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:02.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:02.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:02.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:31:02.769 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:31:02.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:02.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:03.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:04.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:04.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:04.337 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:31:04.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:31:04.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:04.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:04.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:05.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:05.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:05.797 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:31:05.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:31:05.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:05.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:06.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:07.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:07.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:07.338 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:31:07.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:31:07.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:07.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:07.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:08.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:08.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:08.888 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:31:08.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:31:08.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:08.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:09.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:10.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:10.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:10.364 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:31:10.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:31:10.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:10.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:10.722 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/409695429,v1:127.0.0.1:6819/409695429] [v2:127.0.0.1:6820/409695429,v1:127.0.0.1:6821/409695429] exists,up e636a31a-b223-4902-beab-826a6cd4b5d7 2024-02-05T05:31:10.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:31:10.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:31:10.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:31:10.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:31:10.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-02-05T05:31:10.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:31:10.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:31:10.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T05:31:10.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:31:10.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:31:10.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:31:10.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:31:10.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:31:10.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:31:10.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:31:10.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:31:10.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:31:10.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:31:10.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:31:10.728 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:31:10.728 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:31:10.728 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:31:10.728 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:31:10.728 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:31:10.728 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:31:10.728 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:31:10.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:31:10.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:31:10.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:31:10.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:31:10.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:31:10.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:31:10.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:31:10.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:31:10.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:31:10.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:31:10.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:31:10.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:31:10.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:31:10.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:31:10.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0282ec22-c7d3-4fbd-9dce-8fb6471fec13 2024-02-05T05:31:10.731 INFO:tasks.workunit.client.0.smithi148.stdout:add osd3 0282ec22-c7d3-4fbd-9dce-8fb6471fec13 2024-02-05T05:31:10.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 0282ec22-c7d3-4fbd-9dce-8fb6471fec13' 2024-02-05T05:31:10.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:31:10.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCecsBlVIDPLBAAkh88HOtep3Mdg26LT0Q+MQ== 2024-02-05T05:31:10.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCecsBlVIDPLBAAkh88HOtep3Mdg26LT0Q+MQ=="}' 2024-02-05T05:31:10.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0282ec22-c7d3-4fbd-9dce-8fb6471fec13 -i td/osd-scrub-repair/3/new.json 2024-02-05T05:31:11.129 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:31:11.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-02-05T05:31:11.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCecsBlVIDPLBAAkh88HOtep3Mdg26LT0Q+MQ== --osd-uuid 0282ec22-c7d3-4fbd-9dce-8fb6471fec13 2024-02-05T05:31:11.184 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:31:11.183+0000 7f9e04879640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:31:11.231 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:31:11.227+0000 7f9e04879640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:31:11.233 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:31:11.231+0000 7f9e04879640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:31:11.233 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:31:11.231+0000 7f9e04879640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:31:11.233 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:31:11.231+0000 7f9e04879640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:31:11.233 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:31:11.231+0000 7f9e04879640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:31:11.310 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:31:11.307+0000 7f9e04879640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-02-05T05:31:18.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-02-05T05:31:18.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:31:18.634 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd3 key to auth repository 2024-02-05T05:31:18.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T05:31:18.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:31:19.013 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.3 2024-02-05T05:31:19.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T05:31:19.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:31:19.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:31:19.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:31:19.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:31:19.045 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:31:19.043+0000 7f5b4e6e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:31:19.047 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:31:19.043+0000 7f5b4e6e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:31:19.048 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:31:19.047+0000 7f5b4e6e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:31:19.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T05:31:19.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:31:19.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:31:19.343 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:31:19.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:31:19.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:31:19.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:19.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:31:19.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:19.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:19.607 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:31:19.603+0000 7f5b4e6e5640 -1 Falling back to public interface 2024-02-05T05:31:19.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:20.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:20.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:20.690 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:31:20.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:31:20.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:20.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:21.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:22.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:22.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:22.008 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:31:22.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:31:22.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:22.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:22.274 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:31:22.271+0000 7f5b4e6e5640 -1 osd.3 0 log_to_monitors true 2024-02-05T05:31:22.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:23.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:23.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:23.323 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:31:23.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:31:23.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:23.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:23.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:24.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:24.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:24.624 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:31:24.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:31:24.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:24.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:25.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:26.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:26.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:26.029 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:31:26.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:31:26.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:26.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:26.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:27.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:27.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:27.492 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:31:27.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:31:27.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:27.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:27.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:28.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:28.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:28.804 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:31:28.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:31:28.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:28.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:29.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:30.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:30.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:30.206 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:31:30.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:31:30.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:30.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:30.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:31.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:31.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:31.574 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:31:31.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:31:31.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:31.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:31.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:32.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:32.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:32.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:31:32.955 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:31:32.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:32.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:33.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:34.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:34.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:34.380 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:31:34.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:31:34.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:34.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:34.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:35.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:35.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:35.847 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:31:35.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:31:35.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:35.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:36.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:37.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:37.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:37.237 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:31:37.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:31:37.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:37.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:37.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:38.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:38.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:38.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:31:38.824 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:31:38.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:38.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:39.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:40.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:40.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:40.476 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:31:40.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:31:40.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:40.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:41.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:42.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:42.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:42.119 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:31:42.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:31:42.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:42.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:42.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:43.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:43.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:43.850 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:31:43.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:31:43.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:43.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:44.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:45.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:45.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:45.675 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:31:45.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:31:45.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:45.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:46.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:47.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:47.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:47.033 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:31:47.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:31:47.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:47.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:47.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:48.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:48.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:48.583 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:31:48.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:31:48.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:48.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:48.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:49.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:49.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:49.910 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T05:31:49.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T05:31:49.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:49.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:31:50.231 INFO:tasks.workunit.client.0.smithi148.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1225878055,v1:127.0.0.1:6827/1225878055] [v2:127.0.0.1:6828/1225878055,v1:127.0.0.1:6829/1225878055] exists,up 0282ec22-c7d3-4fbd-9dce-8fb6471fec13 2024-02-05T05:31:50.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:31:50.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:31:50.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:31:50.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-02-05T05:31:50.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:31:50.537 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T05:31:50.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:31:50.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:31:51.023 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T05:31:51.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:31:52.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:31:54.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-02-05T05:31:54.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:31:54.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:31:54.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:31:54.845 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:31:54.845 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:31:54.846 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:31:54.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:31:54.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:31:54.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:31:54.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:31:54.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:31:54.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:31:54.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:31:54.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:31:54.974 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:31:55.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:31:55.394 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:31:55.394 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:31:55.394 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T05:31:55.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:31:55.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:55.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:31:55.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-02-05T05:31:55.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-02-05T05:31:55.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-02-05T05:31:55.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:55.570 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:31:55.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-02-05T05:31:55.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-02-05T05:31:55.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672983' 2024-02-05T05:31:55.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:55.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:31:55.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-02-05T05:31:55.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-02-05T05:31:55.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672983 2-64424509460' 2024-02-05T05:31:55.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:55.875 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:31:56.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345931 2024-02-05T05:31:56.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345931 2024-02-05T05:31:56.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672983 2-64424509460 3-85899345931' 2024-02-05T05:31:56.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:56.012 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-02-05T05:31:56.012 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:56.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:31:56.013 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-02-05T05:31:56.013 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:56.014 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836510 2024-02-05T05:31:56.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-02-05T05:31:56.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-02-05T05:31:56.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:31:56.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-02-05T05:31:56.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:56.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-02-05T05:31:56.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:56.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:31:56.296 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-02-05T05:31:56.296 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:56.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-02-05T05:31:56.298 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672983 2024-02-05T05:31:56.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-02-05T05:31:56.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:31:56.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-02-05T05:31:56.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:56.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-02-05T05:31:56.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:56.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:31:56.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-02-05T05:31:56.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:56.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-02-05T05:31:56.623 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509460 2024-02-05T05:31:56.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-02-05T05:31:56.623 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:31:56.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509460 2024-02-05T05:31:56.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:31:57.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:31:57.962 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:31:58.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509460 2024-02-05T05:31:58.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:58.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345931 2024-02-05T05:31:58.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:58.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:31:58.285 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345931 2024-02-05T05:31:58.285 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:58.286 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.3 seq 85899345931 2024-02-05T05:31:58.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345931 2024-02-05T05:31:58.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-02-05T05:31:58.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:31:58.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345931 -lt 85899345931 2024-02-05T05:31:58.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:31:58.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:31:58.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:31:58.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:31:58.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:31:58.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:31:58.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:31:58.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:31:58.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:31:58.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:31:58.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:31:59.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:31:59.315 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:31:59.316 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:31:59.316 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:31:59.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:31:59.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:31:59.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:31:59.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-02-05T05:31:59.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:31:59.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:31:59.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-02-05T05:31:59.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:31:59.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-02-05T05:32:00.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:32:00.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:32:00.944 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'ecpool' created 2024-02-05T05:32:00.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:32:01.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-02-05T05:32:01.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-02-05T05:32:02.621 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 2 allow_ec_overwrites to true 2024-02-05T05:32:02.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:32:02.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:32:02.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:32:02.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:32:02.647 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:32:02.648 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:32:02.648 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:32:02.648 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:32:02.648 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:32:02.649 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:32:02.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:32:02.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:32:02.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:32:02.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:32:02.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:32:02.773 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:32:03.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:32:03.082 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:32:03.082 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:32:03.082 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T05:32:03.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:32:03.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:03.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:32:03.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-02-05T05:32:03.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-02-05T05:32:03.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-02-05T05:32:03.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:03.268 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:32:03.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-02-05T05:32:03.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-02-05T05:32:03.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672985' 2024-02-05T05:32:03.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:03.410 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:32:03.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-02-05T05:32:03.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-02-05T05:32:03.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672985 2-64424509463' 2024-02-05T05:32:03.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:03.550 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:32:03.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345933 2024-02-05T05:32:03.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345933 2024-02-05T05:32:03.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672985 2-64424509463 3-85899345933' 2024-02-05T05:32:03.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:03.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-02-05T05:32:03.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:03.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:32:03.691 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:03.691 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-02-05T05:32:03.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-02-05T05:32:03.692 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836512 2024-02-05T05:32:03.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-02-05T05:32:03.693 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:03.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-02-05T05:32:03.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:03.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-02-05T05:32:03.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:03.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:32:03.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-02-05T05:32:03.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:03.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-02-05T05:32:03.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-02-05T05:32:03.989 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672985 2024-02-05T05:32:03.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:32:04.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672985 -lt 42949672985 2024-02-05T05:32:04.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:04.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:04.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-02-05T05:32:04.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:32:04.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:04.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-02-05T05:32:04.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-02-05T05:32:04.304 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509463 2024-02-05T05:32:04.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-02-05T05:32:04.305 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:32:04.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-02-05T05:32:04.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:04.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345933 2024-02-05T05:32:04.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:04.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:32:04.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:04.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345933 2024-02-05T05:32:04.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345933 2024-02-05T05:32:04.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2024-02-05T05:32:04.641 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.3 seq 85899345933 2024-02-05T05:32:04.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:32:04.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345933 -lt 85899345933 2024-02-05T05:32:04.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:32:04.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:04.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:05.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:32:05.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:32:05.329 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:32:05.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:32:05.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:32:05.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:32:05.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:32:05.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:32:05.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:32:05.657 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:32:05.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:05.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:06.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:32:06.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:32:06.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:32:06.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:32:06.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-02-05T05:32:06.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T05:32:06.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-02-05T05:32:06.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-02-05T05:32:06.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:32:06.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:32:06.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T05:32:06.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:32:06.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:32:06.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:32:06.463 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T05:32:06.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:32:07.040 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T05:32:07.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:32:07.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:32:07.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:32:07.356 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-02-05T05:32:07.356 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T05:32:07.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:32:07.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:32:07.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:32:07.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-02-05T05:32:07.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-02-05T05:32:07.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-02-05T05:32:07.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T05:32:07.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-02-05T05:32:07.665 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:32:07.665 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:32:07.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-02-05T05:32:07.976 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:32:07.976 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:32:07.976 INFO:tasks.workunit.client.0.smithi148.stderr:0' 2024-02-05T05:32:07.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-02-05T05:32:07.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-02-05T05:32:07.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-02-05T05:32:07.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-02-05T05:32:07.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-02-05T05:32:07.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-02-05T05:32:07.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:32:07.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:32:07.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-02-05T05:32:07.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:32:07.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:32:07.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:32:07.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:32:07.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:32:07.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:32:07.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:32:07.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:32:07.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:32:07.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:32:07.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:32:07.980 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:32:07.980 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:32:07.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:32:07.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:32:07.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:32:08.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:32:08.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:32:08.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:32:08.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:32:08.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:32:08.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:32:08.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:32:08.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:32:10.702 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:32:11.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:32:11.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:32:11.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:32:11.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:32:11.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:32:11.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:32:11.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:32:11.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:32:11.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:32:11.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:32:11.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:32:11.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:32:11.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:32:11.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:32:11.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:32:11.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:32:11.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:32:11.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:32:11.241 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:32:11.241 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:32:11.241 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:32:11.241 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:32:11.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:32:11.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:32:11.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:32:11.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:32:11.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:32:11.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:32:11.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:32:11.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:32:11.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:32:11.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:32:11.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:32:11.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:32:11.242 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.3 2024-02-05T05:32:11.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:32:11.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:32:11.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:32:11.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:32:11.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:32:11.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:32:11.277 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:11.278+0000 7f75d4d6a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:11.280 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:11.278+0000 7f75d4d6a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:11.281 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:11.282+0000 7f75d4d6a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:11.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:32:11.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:32:11.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:32:11.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:32:11.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:32:11.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:11.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:32:11.563 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:32:11.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:11.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:32:11.814 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:11.814+0000 7f75d4d6a640 -1 Falling back to public interface 2024-02-05T05:32:11.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:12.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:12.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:12.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:32:12.917 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:32:12.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:12.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:32:13.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:14.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:14.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:14.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:32:14.229 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:32:14.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:14.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:32:14.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:14.586 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:14.570+0000 7f75d4d6a640 -1 osd.3 38 log_to_monitors true 2024-02-05T05:32:15.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:15.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:15.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:32:15.576 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:32:15.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:15.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:32:16.240 INFO:tasks.workunit.client.0.smithi148.stderr:osd.3 up in weight 1 up_from 43 up_thru 43 down_at 39 last_clean_interval [20,38) [v2:127.0.0.1:6826/3033577145,v1:127.0.0.1:6827/3033577145] [v2:127.0.0.1:6828/3033577145,v1:127.0.0.1:6829/3033577145] exists,up 0282ec22-c7d3-4fbd-9dce-8fb6471fec13 2024-02-05T05:32:16.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:32:16.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:32:16.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:32:16.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:32:16.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:32:16.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:32:16.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:32:16.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:32:16.242 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:32:16.242 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:32:16.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:32:16.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:32:16.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:32:16.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:32:16.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:32:16.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:32:16.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:32:16.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:32:16.362 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:32:16.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:32:16.946 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:32:16.946 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:32:16.946 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T05:32:16.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:32:16.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:16.947 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:32:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-02-05T05:32:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-02-05T05:32:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-02-05T05:32:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:17.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:32:17.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672989 2024-02-05T05:32:17.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672989 2024-02-05T05:32:17.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672989' 2024-02-05T05:32:17.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:17.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:32:17.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-02-05T05:32:17.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-02-05T05:32:17.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672989 2-64424509467' 2024-02-05T05:32:17.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:17.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:32:17.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593730 2024-02-05T05:32:17.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593730 2024-02-05T05:32:17.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672989 2-64424509467 3-184683593730' 2024-02-05T05:32:17.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:17.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-02-05T05:32:17.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:17.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:32:17.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-02-05T05:32:17.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:17.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-02-05T05:32:17.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-02-05T05:32:17.692 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836516 2024-02-05T05:32:17.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:18.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836516 2024-02-05T05:32:18.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:18.002 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672989 2024-02-05T05:32:18.003 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:18.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:32:18.005 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672989 2024-02-05T05:32:18.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:18.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672989 2024-02-05T05:32:18.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-02-05T05:32:18.007 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672989 2024-02-05T05:32:18.007 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:32:18.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672989 -lt 42949672989 2024-02-05T05:32:18.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:18.312 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-02-05T05:32:18.312 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:18.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:32:18.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-02-05T05:32:18.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:18.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-02-05T05:32:18.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-02-05T05:32:18.317 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509467 2024-02-05T05:32:18.317 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:32:18.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-02-05T05:32:18.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:18.633 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-184683593730 2024-02-05T05:32:18.633 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:18.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:32:18.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-184683593730 2024-02-05T05:32:18.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:18.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593730 2024-02-05T05:32:18.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2024-02-05T05:32:18.637 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 184683593730 2024-02-05T05:32:18.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:32:18.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593730 -lt 184683593730 2024-02-05T05:32:18.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:32:18.984 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:18.984 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:19.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:32:19.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:32:19.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:32:19.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:32:19.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:32:19.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:32:19.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:32:19.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:32:19.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:32:19.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:32:19.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:19.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:20.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:32:20.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T05:32:20.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:32:20.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:32:20.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:32:20.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:32:20.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:32:20.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:32:20.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:32:20.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:32:20.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:32:20.134 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:32:20.134 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:32:20.134 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:32:20.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:32:20.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:32:20.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:20.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:20.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:32:20.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:32:20.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:32:20.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:32:20.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:32:20.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:32:20.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:32:20.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:32:20.801 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:32:21.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:32:21.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:32:21.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:32:21.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:32:21.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:32:21.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:32:21.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:32:21.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:32:21.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:32:21.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:32:21.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:32:21.398 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:32:21.398 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:32:21.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:32:21.705 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:32:21.705 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:21.705 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:22.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:32:22.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:32:22.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:32:22.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:32:22.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:32:22.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:32:22.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:32:22.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:32:22.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:32:22.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:32:22.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:32:22.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T05:32:22.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:32:22.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T05:32:22.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:32:22.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:32:22.838 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:32:22.838 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:32:22.839 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:32:22.839 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:32:22.839 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:32:22.839 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:32:23.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:32:23.130 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:32:23.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:23.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:23.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:32:23.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:32:23.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:32:23.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:32:23.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:32:23.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:32:23.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:32:23.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:32:23.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:32:23.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:32:23.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:32:23.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:32:23.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:32:23.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:32:23.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:32:23.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:32:24.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:32:00.777918+0000 2024-02-05T05:32:24.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:32:24.435 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:32:24.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:32:00.777918+0000 2024-02-05T05:32:24.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:32:24.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:32:00.777918+0000 2024-02-05T05:32:24.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:32:24.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:32:24.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:32:24.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:32:24.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:32:24.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:32:24.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:32:24.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:32:24.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:32:00.777918+0000 '>' 2024-02-05T05:32:00.777918+0000 2024-02-05T05:32:24.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:32:25.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:32:25.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:32:25.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:32:25.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:32:25.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:32:25.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:32:25.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:32:26.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:32:00.777918+0000 '>' 2024-02-05T05:32:00.777918+0000 2024-02-05T05:32:26.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:32:27.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:32:27.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:32:27.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:32:27.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:32:27.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:32:27.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:32:27.048 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:32:27.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:32:00.777918+0000 '>' 2024-02-05T05:32:00.777918+0000 2024-02-05T05:32:27.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:32:28.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:32:28.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:32:28.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:32:28.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:32:28.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:32:28.359 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:32:28.359 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:32:28.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:32:00.777918+0000 '>' 2024-02-05T05:32:00.777918+0000 2024-02-05T05:32:28.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:32:29.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:32:29.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:32:29.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:32:29.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:32:29.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:32:29.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:32:29.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:32:29.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:32:00.777918+0000 '>' 2024-02-05T05:32:00.777918+0000 2024-02-05T05:32:29.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:32:30.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:32:30.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:32:30.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:32:30.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:32:30.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:32:30.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:32:30.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:32:31.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:32:00.777918+0000 '>' 2024-02-05T05:32:00.777918+0000 2024-02-05T05:32:31.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:32:32.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:32:32.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:32:32.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:32:32.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:32:32.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:32:32.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:32:32.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:32:32.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:32:25.467578+0000 '>' 2024-02-05T05:32:00.777918+0000 2024-02-05T05:32:32.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:32:32.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:32:32.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:32:32.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:32:32.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:32:32.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:32:32.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:32:32.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:32:32.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:32:32.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:32:32.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:32:32.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:32:32.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:32:32.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:32:32.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:32:32.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:32:32.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:32:33.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:32:33.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:32:33.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:32:33.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:32:33.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:32:33.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:32:33.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:32:33.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:32:35.305 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:32:35.305 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:32:35.305 INFO:tasks.workunit.client.0.smithi148.stdout:hinfo_key 2024-02-05T05:32:35.305 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:32:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:32:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:32:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:32:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:32:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:32:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:32:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:32:35.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:32:35.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:32:35.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:32:35.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:32:35.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:32:35.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:32:35.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:32:35.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:32:35.861 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:32:35.861 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:32:35.861 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:32:35.863 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:32:35.863 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:32:35.863 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:32:35.863 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:32:35.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:32:35.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:32:35.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:32:35.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:32:35.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:32:35.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:32:35.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:32:35.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:32:35.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:32:35.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:32:35.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:32:35.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:32:35.864 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.3 2024-02-05T05:32:35.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:32:35.865 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:32:35.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:32:35.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:32:35.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:32:35.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:32:35.899 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:35.897+0000 7f47c5acb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:35.902 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:35.901+0000 7f47c5acb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:35.904 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:35.901+0000 7f47c5acb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:36.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:32:36.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:32:36.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:32:36.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:32:36.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:32:36.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:36.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:32:36.215 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:32:36.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:36.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:32:36.462 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:36.461+0000 7f47c5acb640 -1 Falling back to public interface 2024-02-05T05:32:36.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:37.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:37.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:37.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:32:37.633 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:32:37.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:37.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:32:37.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:38.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:38.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:38.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:32:38.954 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:32:38.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:38.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:32:39.117 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:39.117+0000 7f47c5acb640 -1 osd.3 45 log_to_monitors true 2024-02-05T05:32:39.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:40.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:40.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:40.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:32:40.262 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:32:40.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:40.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:32:40.637 INFO:tasks.workunit.client.0.smithi148.stderr:osd.3 up in weight 1 up_from 49 up_thru 49 down_at 46 last_clean_interval [43,45) [v2:127.0.0.1:6826/2946789054,v1:127.0.0.1:6827/2946789054] [v2:127.0.0.1:6828/2946789054,v1:127.0.0.1:6829/2946789054] exists,up 0282ec22-c7d3-4fbd-9dce-8fb6471fec13 2024-02-05T05:32:40.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:32:40.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:32:40.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:32:40.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:32:40.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:32:40.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:32:40.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:32:40.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:32:40.638 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:32:40.638 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:32:40.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:32:40.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:32:40.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:32:40.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:32:40.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:32:40.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:32:40.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:32:40.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:32:40.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:32:41.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:32:41.083 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:32:41.083 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:32:41.083 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T05:32:41.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:32:41.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:41.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:32:41.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836522 2024-02-05T05:32:41.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836522 2024-02-05T05:32:41.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522' 2024-02-05T05:32:41.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:41.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:32:41.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672995 2024-02-05T05:32:41.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672995 2024-02-05T05:32:41.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-42949672995' 2024-02-05T05:32:41.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:41.389 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:32:41.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509473 2024-02-05T05:32:41.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509473 2024-02-05T05:32:41.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-42949672995 2-64424509473' 2024-02-05T05:32:41.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:41.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:32:41.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397506 2024-02-05T05:32:41.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397506 2024-02-05T05:32:41.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-42949672995 2-64424509473 3-210453397506' 2024-02-05T05:32:41.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:41.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836522 2024-02-05T05:32:41.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:41.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:32:41.673 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836522 2024-02-05T05:32:41.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:41.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836522 2024-02-05T05:32:41.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-02-05T05:32:41.675 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836522 2024-02-05T05:32:41.676 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:41.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836522 2024-02-05T05:32:41.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:32:42.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:32:42.970 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:43.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836522 -lt 21474836522 2024-02-05T05:32:43.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:43.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672995 2024-02-05T05:32:43.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:43.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:32:43.292 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672995 2024-02-05T05:32:43.292 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:43.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672995 2024-02-05T05:32:43.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2024-02-05T05:32:43.294 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672995 2024-02-05T05:32:43.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:32:43.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672995 -lt 42949672995 2024-02-05T05:32:43.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:43.630 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509473 2024-02-05T05:32:43.631 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:43.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:32:43.633 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509473 2024-02-05T05:32:43.633 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:43.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509473 2024-02-05T05:32:43.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-02-05T05:32:43.634 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509473 2024-02-05T05:32:43.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:32:43.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509473 -lt 64424509473 2024-02-05T05:32:43.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:43.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:43.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-210453397506 2024-02-05T05:32:43.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:32:43.974 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-210453397506 2024-02-05T05:32:43.974 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:43.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397506 2024-02-05T05:32:43.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 210453397506' 2024-02-05T05:32:43.975 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 210453397506 2024-02-05T05:32:43.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:32:44.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397506 -lt 210453397506 2024-02-05T05:32:44.307 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:32:44.308 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:44.308 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:44.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:32:44.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:32:44.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:32:44.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:32:44.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:32:44.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:32:44.715 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:32:44.715 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:32:45.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:32:45.027 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:32:45.027 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:45.028 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:45.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:32:45.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:32:45.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:32:45.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:32:45.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:32:45.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-02-05T05:32:45.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:32:45.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:32:45.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-02-05T05:32:45.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:32:45.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:32:45.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:32:45.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:32:45.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:32:45.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:32:45.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:32:45.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:32:45.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:32:45.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:32:45.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:32:45.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:32:45.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:32:45.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:32:45.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:32:45.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:32:45.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:32:45.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:32:45.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:32:45.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:32:45.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:32:45.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:32:45.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:32:45.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:32:48.126 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:32:48.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:32:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:32:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:32:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:32:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:32:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:32:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:32:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:32:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:32:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:32:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:32:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:32:48.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:32:48.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:32:48.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:32:48.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:32:48.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:32:48.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:32:48.670 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:32:48.670 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:32:48.670 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:32:48.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:32:48.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:32:48.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:32:48.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:32:48.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:32:48.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:32:48.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:32:48.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:32:48.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:32:48.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:32:48.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:32:48.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:32:48.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:32:48.673 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:32:48.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:32:48.673 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:32:48.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:32:48.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:32:48.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:32:48.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:32:48.706 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:48.705+0000 7f5bbaeca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:48.709 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:48.709+0000 7f5bbaeca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:48.710 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:48.709+0000 7f5bbaeca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:48.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:32:48.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:32:48.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:32:48.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:32:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:32:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:32:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:32:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:32:49.243 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:49.241+0000 7f5bbaeca640 -1 Falling back to public interface 2024-02-05T05:32:49.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:50.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:50.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:50.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:32:50.318 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:32:50.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:50.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:32:50.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:51.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:51.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:51.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:32:51.642 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:32:51.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:51.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:32:51.940 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:32:51.941+0000 7f5bbaeca640 -1 osd.1 50 log_to_monitors true 2024-02-05T05:32:51.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:52.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:52.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:52.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:32:52.976 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:32:52.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:52.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:32:53.530 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 51 last_clean_interval [10,50) [v2:127.0.0.1:6810/4274372029,v1:127.0.0.1:6811/4274372029] [v2:127.0.0.1:6812/4274372029,v1:127.0.0.1:6813/4274372029] exists,up adc22c43-3a38-4d23-933d-b1fb1e5ffc07 2024-02-05T05:32:53.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:32:53.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:32:53.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:32:53.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:32:53.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:32:53.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:32:53.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:32:53.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:32:53.531 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:32:53.531 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:32:53.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:32:53.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:32:53.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:32:53.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:32:53.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:32:53.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:32:53.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:32:53.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:32:53.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:32:54.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:32:54.151 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:32:54.151 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:32:54.151 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T05:32:54.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:32:54.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:54.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:32:54.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836525 2024-02-05T05:32:54.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836525 2024-02-05T05:32:54.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525' 2024-02-05T05:32:54.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:54.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:32:54.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201282 2024-02-05T05:32:54.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201282 2024-02-05T05:32:54.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-236223201282' 2024-02-05T05:32:54.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:54.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:32:54.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509476 2024-02-05T05:32:54.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509476 2024-02-05T05:32:54.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-236223201282 2-64424509476' 2024-02-05T05:32:54.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:54.604 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:32:54.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397510 2024-02-05T05:32:54.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397510 2024-02-05T05:32:54.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-236223201282 2-64424509476 3-210453397510' 2024-02-05T05:32:54.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:54.751 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836525 2024-02-05T05:32:54.751 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:54.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:32:54.753 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836525 2024-02-05T05:32:54.753 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:54.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836525 2024-02-05T05:32:54.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-02-05T05:32:54.754 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836525 2024-02-05T05:32:54.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:55.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836524 -lt 21474836525 2024-02-05T05:32:55.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:32:56.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:32:56.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:56.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836526 -lt 21474836525 2024-02-05T05:32:56.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:56.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-236223201282 2024-02-05T05:32:56.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:56.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:32:56.366 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-236223201282 2024-02-05T05:32:56.367 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:56.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201282 2024-02-05T05:32:56.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 236223201282' 2024-02-05T05:32:56.368 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 236223201282 2024-02-05T05:32:56.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:32:56.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201282 -lt 236223201282 2024-02-05T05:32:56.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:56.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509476 2024-02-05T05:32:56.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:56.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:32:56.704 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509476 2024-02-05T05:32:56.704 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:56.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509476 2024-02-05T05:32:56.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2024-02-05T05:32:56.705 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509476 2024-02-05T05:32:56.706 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:32:57.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509476 -lt 64424509476 2024-02-05T05:32:57.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:57.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:57.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-210453397510 2024-02-05T05:32:57.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:32:57.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-210453397510 2024-02-05T05:32:57.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:57.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397510 2024-02-05T05:32:57.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 210453397510' 2024-02-05T05:32:57.018 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 210453397510 2024-02-05T05:32:57.018 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:32:57.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397510 -lt 210453397510 2024-02-05T05:32:57.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:32:57.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:57.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:57.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:32:57.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:32:57.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:32:57.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:32:57.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:32:57.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:32:57.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:32:57.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:32:58.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:32:58.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:32:58.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:58.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:58.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:32:58.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:32:58.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:32:58.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:32:58.442 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:32:58.442 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:32:58.442 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:32:58.442 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:32:58.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:32:58.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:32:58.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:32:58.767 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:32:58.767 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:32:58.767 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:32:58.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:32:58.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:32:59.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:32:25.467578+0000 2024-02-05T05:32:59.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:32:59.397 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:32:59.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:32:25.467578+0000 2024-02-05T05:32:59.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:32:59.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:32:25.467578+0000 2024-02-05T05:32:59.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:32:59.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:32:59.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:32:59.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:32:59.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:32:59.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:32:59.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:32:59.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:32:59.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:32:25.467578+0000 '>' 2024-02-05T05:32:25.467578+0000 2024-02-05T05:32:59.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:00.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:00.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:00.744 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:33:00.744 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:33:00.744 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:00.745 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:00.745 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:33:01.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:32:25.467578+0000 '>' 2024-02-05T05:32:25.467578+0000 2024-02-05T05:33:01.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:02.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:02.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:02.060 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:33:02.060 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:33:02.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:02.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:02.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:33:02.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:32:25.467578+0000 '>' 2024-02-05T05:32:25.467578+0000 2024-02-05T05:33:02.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:03.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:03.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:03.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:33:03.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:33:03.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:03.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:03.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:33:03.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:32:25.467578+0000 '>' 2024-02-05T05:32:25.467578+0000 2024-02-05T05:33:03.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:04.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:04.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:04.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:33:04.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:33:04.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:04.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:04.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:33:05.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:32:25.467578+0000 '>' 2024-02-05T05:32:25.467578+0000 2024-02-05T05:33:05.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:06.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:06.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:06.005 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:33:06.005 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:33:06.005 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:06.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:06.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:33:06.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:33:00.091013+0000 '>' 2024-02-05T05:32:25.467578+0000 2024-02-05T05:33:06.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:33:06.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:33:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:33:06.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:33:06.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:33:06.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:33:06.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:33:06.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:33:06.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:33:06.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:33:06.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:33:08.683 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:33:08.684 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:33:08.684 INFO:tasks.workunit.client.0.smithi148.stdout:hinfo_key 2024-02-05T05:33:08.684 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:33:08.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:33:08.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:33:08.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:33:08.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:33:08.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:33:08.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:33:08.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:33:08.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:33:08.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:33:08.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:33:08.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:33:08.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:33:08.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:33:08.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:33:08.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:33:08.981 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:33:08.981 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:33:08.981 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:33:08.981 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:33:08.981 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:33:08.981 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:33:08.981 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:33:08.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:33:08.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:33:08.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:33:08.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:33:08.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:33:08.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:33:08.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:33:08.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:33:08.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:33:08.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:33:08.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:33:08.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:33:08.983 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.3 2024-02-05T05:33:08.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:33:08.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:33:08.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:33:08.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:33:08.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:33:08.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:33:09.016 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:33:09.013+0000 7f0158a9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:09.018 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:33:09.017+0000 7f0158a9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:09.019 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:33:09.017+0000 7f0158a9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:09.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:33:09.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:33:09.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:33:09.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:33:09.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:33:09.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:09.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:33:09.291 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:33:09.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:09.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:33:09.562 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:33:09.561+0000 7f0158a9c640 -1 Falling back to public interface 2024-02-05T05:33:09.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:10.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:10.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:10.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:33:10.631 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:33:10.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:10.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:33:10.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:11.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:11.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:11.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:33:11.933 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:33:11.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:11.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:33:12.199 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:33:12.196+0000 7f0158a9c640 -1 osd.3 57 log_to_monitors true 2024-02-05T05:33:12.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:13.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:13.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:13.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:33:13.288 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:33:13.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:13.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:33:13.699 INFO:tasks.workunit.client.0.smithi148.stderr:osd.3 up in weight 1 up_from 62 up_thru 62 down_at 58 last_clean_interval [49,57) [v2:127.0.0.1:6826/1913424712,v1:127.0.0.1:6827/1913424712] [v2:127.0.0.1:6828/1913424712,v1:127.0.0.1:6829/1913424712] exists,up 0282ec22-c7d3-4fbd-9dce-8fb6471fec13 2024-02-05T05:33:13.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:33:13.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:33:13.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:33:13.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:33:13.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:33:13.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:33:13.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:33:13.700 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:33:13.701 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:33:13.701 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:33:13.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:33:13.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:33:13.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:33:13.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:33:13.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:33:13.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:33:13.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:33:13.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:33:13.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:33:14.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:33:14.337 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:33:14.337 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:33:14.337 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T05:33:14.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:33:14.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:14.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:33:14.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836530 2024-02-05T05:33:14.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836530 2024-02-05T05:33:14.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530' 2024-02-05T05:33:14.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:14.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201287 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201287 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-236223201287' 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:33:14.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509481 2024-02-05T05:33:14.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509481 2024-02-05T05:33:14.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-236223201287 2-64424509481' 2024-02-05T05:33:14.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:14.801 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:33:14.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972354 2024-02-05T05:33:14.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972354 2024-02-05T05:33:14.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-236223201287 2-64424509481 3-266287972354' 2024-02-05T05:33:14.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:14.943 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836530 2024-02-05T05:33:14.943 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:14.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:33:14.944 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836530 2024-02-05T05:33:14.944 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:14.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836530 2024-02-05T05:33:14.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-02-05T05:33:14.946 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836530 2024-02-05T05:33:14.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836530 2024-02-05T05:33:15.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:33:16.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:33:16.249 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:16.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836530 2024-02-05T05:33:16.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:16.579 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-236223201287 2024-02-05T05:33:16.579 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:16.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:33:16.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-236223201287 2024-02-05T05:33:16.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:16.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201287 2024-02-05T05:33:16.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 236223201287' 2024-02-05T05:33:16.583 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 236223201287 2024-02-05T05:33:16.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:33:16.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201287 -lt 236223201287 2024-02-05T05:33:16.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:16.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:16.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509481 2024-02-05T05:33:16.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:33:16.883 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509481 2024-02-05T05:33:16.883 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:16.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509481 2024-02-05T05:33:16.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2024-02-05T05:33:16.884 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509481 2024-02-05T05:33:16.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:33:17.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509481 -lt 64424509481 2024-02-05T05:33:17.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:17.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972354 2024-02-05T05:33:17.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:17.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:33:17.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972354 2024-02-05T05:33:17.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:17.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972354 2024-02-05T05:33:17.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972354' 2024-02-05T05:33:17.192 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 266287972354 2024-02-05T05:33:17.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:33:17.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972354 -lt 266287972354 2024-02-05T05:33:17.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:33:17.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:17.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:17.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:33:17.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:33:17.865 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:33:17.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:33:17.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:33:17.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:33:17.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:33:17.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:33:18.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:33:18.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:33:18.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:18.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:18.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:33:18.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:33:18.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:33:18.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:33:18.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:33:18.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-02-05T05:33:18.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:33:18.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:33:18.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-02-05T05:33:18.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-02-05T05:33:18.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:33:18.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:33:18.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:33:18.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:33:18.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 183379"' 2024-02-05T05:33:18.588 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 183379' 2024-02-05T05:33:18.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:33:18.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:33:18.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:33:18.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 183380"' 2024-02-05T05:33:18.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 183380' 2024-02-05T05:33:18.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:33:18.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 183379 183380' 2024-02-05T05:33:18.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:33:18.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:33:18.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:33:18.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 183379 2024-02-05T05:33:18.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/183382: /' 2024-02-05T05:33:18.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:33:18.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/183384: /' 2024-02-05T05:33:22.343 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:33:22.343 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:33:22.343 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:33:22.343 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:33:22.343 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:33:22.343 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:33:22.343 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:33:22.343 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:33:22.343 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:33:22.343 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:33:22.343 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:33:22.344 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:33:22.344 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:33:22.344 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:33:22.344 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:33:22.344 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:33:22.344 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:33:22.344 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:33:22.344 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:33:22.344 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:33:22.344 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:33:22.344 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:33:22.344 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:33:22.344 INFO:tasks.workunit.client.0.smithi148.stderr:183382: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:33:22.345 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:33:22.345 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:33:22.345 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:33:22.345 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:33:22.345 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:33:22.345 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:33:22.345 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:33:22.345 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:33:22.348 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:33:22.348 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:33:22.348 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:33:22.348 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:33:22.348 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:33:22.348 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:33:22.348 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:33:22.349 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:33:22.349 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:33:22.349 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:33:22.349 INFO:tasks.workunit.client.0.smithi148.stderr:183382: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:33:22.349 INFO:tasks.workunit.client.0.smithi148.stderr:183382: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:33:22.349 INFO:tasks.workunit.client.0.smithi148.stderr:183382: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:33:22.349 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:33:22.349 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:33:22.349 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:33:22.349 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:33:22.349 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:33:22.349 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:33:22.349 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:33:22.349 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:33:22.350 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:33:22.350 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:33:22.350 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:33:22.350 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:33:22.350 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:33:22.350 INFO:tasks.workunit.client.0.smithi148.stderr:183382: start osd.1 2024-02-05T05:33:22.350 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:33:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:33:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:33:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T05:33:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:33:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:33:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:33:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:33:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T05:33:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:33:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:33:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:33:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:33:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:33:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:33:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:33:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:33:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:33:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:33:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:33:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T05:33:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:33:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:33:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-02-05T05:33:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:183384: remove 2#2:eb822e21:::SOMETHING:head# 2024-02-05T05:33:22.510 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T05:33:22.510 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:33:22.510 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:33:22.510 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:33:22.510 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:33:22.510 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:33:22.510 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:33:22.510 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:33:22.511 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:33:22.511 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:33:22.511 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:33:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:33:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:33:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:33:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:33:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:33:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:33:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:33:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:183384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:33:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:183384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:33:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:183384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:33:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:33:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:33:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:33:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:33:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:33:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:33:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:33:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:33:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:33:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:33:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:33:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:33:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:33:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:183384: start osd.2 2024-02-05T05:33:22.513 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:33:26.981 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:33:26.981 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:33:26.981 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:33:26.981 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:33:26.981 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:33:26.981 INFO:tasks.workunit.client.0.smithi148.stderr:183382: 2024-02-05T05:33:22.380+0000 7f92ebdf6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:26.982 INFO:tasks.workunit.client.0.smithi148.stderr:183382: 2024-02-05T05:33:22.384+0000 7f92ebdf6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:26.982 INFO:tasks.workunit.client.0.smithi148.stderr:183382: 2024-02-05T05:33:22.384+0000 7f92ebdf6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:26.982 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:33:26.982 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:33:26.982 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:33:26.982 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:33:26.982 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:33:26.982 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:26.982 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:33:26.982 INFO:tasks.workunit.client.0.smithi148.stderr:183382: 0 2024-02-05T05:33:26.982 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:26.982 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:33:26.982 INFO:tasks.workunit.client.0.smithi148.stderr:183382: 2024-02-05T05:33:22.948+0000 7f92ebdf6640 -1 Falling back to public interface 2024-02-05T05:33:26.983 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:26.983 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:26.983 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:26.983 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:33:26.983 INFO:tasks.workunit.client.0.smithi148.stderr:183382: 1 2024-02-05T05:33:26.983 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:26.983 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:33:26.983 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:26.983 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:26.983 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:26.983 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:33:26.983 INFO:tasks.workunit.client.0.smithi148.stderr:183382: 2 2024-02-05T05:33:26.984 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:26.984 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:33:26.984 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:26.984 INFO:tasks.workunit.client.0.smithi148.stderr:183382: 2024-02-05T05:33:25.932+0000 7f92ebdf6640 -1 osd.1 64 log_to_monitors true 2024-02-05T05:33:26.984 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:26.984 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:26.984 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:33:26.984 INFO:tasks.workunit.client.0.smithi148.stderr:183382: 3 2024-02-05T05:33:26.984 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:26.984 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:33:27.148 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:33:27.149 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:33:27.149 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:33:27.149 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:33:27.149 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:33:27.149 INFO:tasks.workunit.client.0.smithi148.stderr:183384: 2024-02-05T05:33:22.540+0000 7fb34b962640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:27.149 INFO:tasks.workunit.client.0.smithi148.stderr:183384: 2024-02-05T05:33:22.548+0000 7fb34b962640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:27.149 INFO:tasks.workunit.client.0.smithi148.stderr:183384: 2024-02-05T05:33:22.548+0000 7fb34b962640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:27.149 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:33:27.149 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:33:27.149 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:33:27.149 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:33:27.150 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:33:27.150 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:27.150 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:33:27.150 INFO:tasks.workunit.client.0.smithi148.stderr:183384: 0 2024-02-05T05:33:27.150 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:27.150 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:33:27.150 INFO:tasks.workunit.client.0.smithi148.stderr:183384: 2024-02-05T05:33:23.088+0000 7fb34b962640 -1 Falling back to public interface 2024-02-05T05:33:27.150 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:27.150 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:27.150 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:27.150 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:33:27.150 INFO:tasks.workunit.client.0.smithi148.stderr:183384: 1 2024-02-05T05:33:27.151 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:27.151 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:33:27.151 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:27.151 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:27.151 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:27.151 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:33:27.151 INFO:tasks.workunit.client.0.smithi148.stderr:183384: 2 2024-02-05T05:33:27.151 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:27.151 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:33:27.151 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:27.151 INFO:tasks.workunit.client.0.smithi148.stderr:183384: 2024-02-05T05:33:26.348+0000 7fb34b962640 -1 osd.2 64 log_to_monitors true 2024-02-05T05:33:27.151 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:27.151 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:27.151 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:33:27.152 INFO:tasks.workunit.client.0.smithi148.stderr:183384: 3 2024-02-05T05:33:27.152 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:27.152 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:33:29.080 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:29.080 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:29.080 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:29.080 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:33:29.080 INFO:tasks.workunit.client.0.smithi148.stderr:183382: 4 2024-02-05T05:33:29.080 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:33:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:183382: osd.1 up in weight 1 up_from 69 up_thru 69 down_at 65 last_clean_interval [55,64) [v2:127.0.0.1:6810/4015526739,v1:127.0.0.1:6811/4015526739] [v2:127.0.0.1:6812/4015526739,v1:127.0.0.1:6813/4015526739] exists,up adc22c43-3a38-4d23-933d-b1fb1e5ffc07 2024-02-05T05:33:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:33:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:33:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:33:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:33:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:33:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:33:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:33:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:33:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:183382: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:33:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:183382: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:33:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:33:29.081 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:33:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:33:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:33:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:33:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:33:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:33:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:33:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:33:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:33:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:183382: 1 2024-02-05T05:33:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:183382: 2 2024-02-05T05:33:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:183382: 3' 2024-02-05T05:33:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:33:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:33:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836534 2024-02-05T05:33:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836534 2024-02-05T05:33:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534' 2024-02-05T05:33:29.083 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:29.404 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hels.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:29.405 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:29.405 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:29.405 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:33:29.405 INFO:tasks.workunit.client.0.smithi148.stderr:183384: 4 2024-02-05T05:33:29.405 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:29.405 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:33:29.405 INFO:tasks.workunit.client.0.smithi148.stderr:183384: osd.2 up in weight 1 up_from 70 up_thru 51 down_at 65 last_clean_interval [15,64) [v2:127.0.0.1:6818/2405306881,v1:127.0.0.1:6819/2405306881] [v2:127.0.0.1:6820/2405306881,v1:127.0.0.1:6821/2405306881] exists,up e636a31a-b223-4902-beab-826a6cd4b5d7 2024-02-05T05:33:29.405 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:33:29.405 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:33:29.405 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:33:29.405 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:33:29.405 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:33:29.406 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:33:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:183384: 1 2024-02-05T05:33:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:183384: 2 2024-02-05T05:33:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:183384: 3' 2024-02-05T05:33:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:33:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:33:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836535 2024-02-05T05:33:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836535 2024-02-05T05:33:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535' 2024-02-05T05:33:29.407 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:30.424 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:33:30.424 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743427 2024-02-05T05:33:30.424 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743427 2024-02-05T05:33:30.424 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-296352743427' 2024-02-05T05:33:30.424 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:30.424 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:33:30.425 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710723 2024-02-05T05:33:30.425 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710723 2024-02-05T05:33:30.425 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-296352743427 2-300647710723' 2024-02-05T05:33:30.425 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:30.425 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:33:30.425 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972359 2024-02-05T05:33:30.425 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972359 2024-02-05T05:33:30.425 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-296352743427 2-300647710723 3-266287972359' 2024-02-05T05:33:30.425 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:30.425 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836535 2024-02-05T05:33:30.425 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:30.425 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:33:30.425 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836535 2024-02-05T05:33:30.426 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:30.426 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836535 2024-02-05T05:33:30.426 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-02-05T05:33:30.426 INFO:tasks.workunit.client.0.smithi148.stderr:183384: waiting osd.0 seq 21474836535 2024-02-05T05:33:30.426 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:30.426 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836535 -lt 21474836535 2024-02-05T05:33:30.426 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:30.426 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-296352743427 2024-02-05T05:33:30.426 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:30.426 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:33:30.426 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-296352743427 2024-02-05T05:33:30.426 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:30.426 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743427 2024-02-05T05:33:30.427 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 296352743427' 2024-02-05T05:33:30.427 INFO:tasks.workunit.client.0.smithi148.stderr:183384: waiting osd.1 seq 296352743427 2024-02-05T05:33:30.427 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:33:31.207 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221pers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:33:31.207 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743426 2024-02-05T05:33:31.207 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743426 2024-02-05T05:33:31.207 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-296352743426' 2024-02-05T05:33:31.207 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:31.207 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:33:31.207 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710722 2024-02-05T05:33:31.208 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710722 2024-02-05T05:33:31.208 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-296352743426 2-300647710722' 2024-02-05T05:33:31.208 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:31.209 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:33:31.209 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972358 2024-02-05T05:33:31.209 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972358 2024-02-05T05:33:31.209 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-296352743426 2-300647710722 3-266287972358' 2024-02-05T05:33:31.209 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:31.210 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836534 2024-02-05T05:33:31.210 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:31.210 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:33:31.210 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836534 2024-02-05T05:33:31.210 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:31.210 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836534 2024-02-05T05:33:31.210 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-02-05T05:33:31.211 INFO:tasks.workunit.client.0.smithi148.stderr:183382: waiting osd.0 seq 21474836534 2024-02-05T05:33:31.211 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:31.211 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836533 -lt 21474836534 2024-02-05T05:33:31.211 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:33:31.211 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:33:31.211 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:31.211 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836535 -lt 21474836534 2024-02-05T05:33:31.211 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:31.211 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-296352743426 2024-02-05T05:33:31.212 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:31.212 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:33:31.212 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-296352743426 2024-02-05T05:33:31.212 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:31.789 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/u4: flush_pg_stats: test 296352743427 -lt 296352743427 2024-02-05T05:33:31.789 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:31.789 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-300647710723 2024-02-05T05:33:31.789 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:31.789 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:33:31.789 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-300647710723 2024-02-05T05:33:31.789 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:31.789 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710723 2024-02-05T05:33:31.789 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 300647710723' 2024-02-05T05:33:31.790 INFO:tasks.workunit.client.0.smithi148.stderr:183384: waiting osd.2 seq 300647710723 2024-02-05T05:33:31.790 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:33:31.790 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710723 -lt 300647710723 2024-02-05T05:33:31.790 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:31.790 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:31.790 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972359 2024-02-05T05:33:31.790 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:33:31.790 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972359 2024-02-05T05:33:31.790 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:31.790 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972359 2024-02-05T05:33:31.791 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972359' 2024-02-05T05:33:31.791 INFO:tasks.workunit.client.0.smithi148.stderr:183384: waiting osd.3 seq 266287972359 2024-02-05T05:33:31.791 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:33:31.791 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972359 -lt 266287972359 2024-02-05T05:33:31.791 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:33:31.791 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:31.791 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:31.791 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:33:31.791 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:33:31.791 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:33:31.791 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:33:31.791 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:33:31.791 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:33:31.792 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:33:31.792 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:33:32.182 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:33:32.183 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:33:32.183 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:32.183 INFO:tasks.workunit.client.0.smithi148.stderr:183384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:32.183 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:33:32.183 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:33:32.183 INFO:tasks.workunit.client.0.smithi148.stderr:183384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:33:32.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:33:32.579 INFO:tasks.workunit.client.0.smithi148.stderr:buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743426 2024-02-05T05:33:32.580 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 296352743426' 2024-02-05T05:33:32.580 INFO:tasks.workunit.client.0.smithi148.stderr:183382: waiting osd.1 seq 296352743426 2024-02-05T05:33:32.580 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:33:32.580 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743427 -lt 296352743426 2024-02-05T05:33:32.580 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:32.580 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-300647710722 2024-02-05T05:33:32.580 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:32.580 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:33:32.580 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-300647710722 2024-02-05T05:33:32.580 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:32.580 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710722 2024-02-05T05:33:32.580 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 300647710722' 2024-02-05T05:33:32.580 INFO:tasks.workunit.client.0.smithi148.stderr:183382: waiting osd.2 seq 300647710722 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710723 -lt 300647710722 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972358 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972358 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972358 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972358' 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: waiting osd.3 seq 266287972358 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972359 -lt 266287972358 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:33:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:32.582 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:32.582 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:33:32.582 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:33:32.582 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:33:32.582 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:33:32.582 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:33:33.288 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:33:33.288 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:183382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:183382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 183380 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:33:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:33:33.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:33:33.290 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:33:33.290 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:33:33.290 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:33:33.290 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:33:33.290 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:33:33.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:33:33.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:33:33.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:33:33.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:33:33.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:33:33.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:33.602 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:33.602 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:33:33.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:33:00.091013+0000 2024-02-05T05:33:33.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:33:34.162 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:33:34.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:33:00.091013+0000 2024-02-05T05:33:34.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:33:34.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:33:00.091013+0000 2024-02-05T05:33:34.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:33:34.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:33:34.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:34.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:33:34.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:33:34.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:34.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:34.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:33:34.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:33:00.091013+0000 '>' 2024-02-05T05:33:00.091013+0000 2024-02-05T05:33:34.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:35.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:35.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:35.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:33:35.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:33:35.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:35.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:35.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:33:35.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:33:00.091013+0000 '>' 2024-02-05T05:33:00.091013+0000 2024-02-05T05:33:35.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:36.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:36.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:36.774 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:33:36.774 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:33:36.774 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:36.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:36.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:33:37.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:33:00.091013+0000 '>' 2024-02-05T05:33:00.091013+0000 2024-02-05T05:33:37.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:38.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:38.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:38.085 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:33:38.086 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:33:38.086 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:38.086 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:38.086 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:33:38.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:33:35.202321+0000 '>' 2024-02-05T05:33:00.091013+0000 2024-02-05T05:33:38.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:33:38.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:33:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:33:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:33:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:33:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 186233"' 2024-02-05T05:33:38.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 186233' 2024-02-05T05:33:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:33:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:33:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:33:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 186235"' 2024-02-05T05:33:38.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 186235' 2024-02-05T05:33:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:33:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 186233 186235' 2024-02-05T05:33:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:33:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:33:38.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 186233 2024-02-05T05:33:38.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:33:38.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/186236: /' 2024-02-05T05:33:38.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:33:38.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/186238: /' 2024-02-05T05:33:41.345 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:33:41.345 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:33:41.345 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T05:33:41.345 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:33:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:33:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:33:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:33:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T05:33:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:33:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:33:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:33:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:33:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:33:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:33:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:33:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:33:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:33:41.347 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:33:41.347 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:33:41.347 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T05:33:41.347 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:33:41.347 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:33:41.347 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-02-05T05:33:41.347 INFO:tasks.workunit.client.0.smithi148.stderr:186238: _ 2024-02-05T05:33:41.347 INFO:tasks.workunit.client.0.smithi148.stderr:186238: hinfo_key 2024-02-05T05:33:41.347 INFO:tasks.workunit.client.0.smithi148.stderr:186238: snapset 2024-02-05T05:33:41.347 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T05:33:41.347 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:33:41.347 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:33:41.347 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:33:41.347 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:33:41.348 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:33:41.348 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:33:41.348 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:33:41.349 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:33:41.349 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:33:41.349 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:33:41.349 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:33:41.349 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:33:41.349 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:33:41.349 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:33:41.349 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:33:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:33:41.351 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:33:41.351 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:33:41.351 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:33:41.351 INFO:tasks.workunit.client.0.smithi148.stderr:186238: start osd.2 2024-02-05T05:33:41.617 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:33:41.617 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:33:41.617 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:33:41.617 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:33:41.618 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:33:41.619 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:33:41.619 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:33:41.619 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:33:41.619 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:33:41.619 INFO:tasks.workunit.client.0.smithi148.stderr:186236: _ 2024-02-05T05:33:41.619 INFO:tasks.workunit.client.0.smithi148.stderr:186236: hinfo_key 2024-02-05T05:33:41.619 INFO:tasks.workunit.client.0.smithi148.stderr:186236: snapset 2024-02-05T05:33:41.619 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:33:41.619 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:33:41.619 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:33:41.619 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:33:41.619 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:33:41.619 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:33:41.619 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:33:41.620 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:33:41.621 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:33:41.621 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:33:41.621 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:33:41.622 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:33:41.622 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:33:41.622 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:33:41.622 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:33:41.622 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:33:41.622 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:33:41.622 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:33:41.622 INFO:tasks.workunit.client.0.smithi148.stderr:186236: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:33:41.622 INFO:tasks.workunit.client.0.smithi148.stderr:186236: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:33:41.622 INFO:tasks.workunit.client.0.smithi148.stderr:186236: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:33:41.622 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:33:41.622 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:33:41.622 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:33:41.622 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:33:41.623 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:33:41.623 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:33:41.623 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:33:41.623 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:33:41.623 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:33:41.624 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:33:41.624 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:33:41.624 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:33:41.624 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:33:41.624 INFO:tasks.workunit.client.0.smithi148.stderr:186236: start osd.1 2024-02-05T05:33:45.999 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:33:45.999 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:33:45.999 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:33:45.999 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:33:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:33:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:33:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:186238: 2024-02-05T05:33:41.380+0000 7f1655416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:186238: 2024-02-05T05:33:41.384+0000 7f1655416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:186238: 2024-02-05T05:33:41.384+0000 7f1655416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:33:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:33:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:33:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:33:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:33:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:33:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:186238: 0 2024-02-05T05:33:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:33:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:186238: 2024-02-05T05:33:41.920+0000 7f1655416640 -1 Falling back to public interface 2024-02-05T05:33:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:33:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:186238: 1 2024-02-05T05:33:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:33:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:46.002 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:33:46.002 INFO:tasks.workunit.client.0.smithi148.stderr:186238: 2 2024-02-05T05:33:46.002 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:46.002 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:33:46.002 INFO:tasks.workunit.client.0.smithi148.stderr:186238: 2024-02-05T05:33:44.640+0000 7f1655416640 -1 osd.2 72 log_to_monitors true 2024-02-05T05:33:46.002 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:46.002 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:46.002 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:46.002 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:33:46.002 INFO:tasks.workunit.client.0.smithi148.stderr:186238: 3 2024-02-05T05:33:46.002 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:46.002 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:33:46.582 INFO:tasks.workunit.client.0.smithi148.stderr:186238: osd.2 up in weight 1 up_from 77 up_thru 77 down_aovery_ops 2024-02-05T05:33:46.582 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:33:46.582 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:33:46.582 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:33:46.582 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:33:46.582 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:33:46.582 INFO:tasks.workunit.client.0.smithi148.stderr:186236: 2024-02-05T05:33:41.652+0000 7f59d7924640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:46.582 INFO:tasks.workunit.client.0.smithi148.stderr:186236: 2024-02-05T05:33:41.652+0000 7f59d7924640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:46.582 INFO:tasks.workunit.client.0.smithi148.stderr:186236: 2024-02-05T05:33:41.656+0000 7f59d7924640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:46.582 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:33:46.582 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:33:46.583 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:33:46.583 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:33:46.583 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:33:46.583 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:46.583 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:33:46.583 INFO:tasks.workunit.client.0.smithi148.stderr:186236: 0 2024-02-05T05:33:46.583 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:46.583 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:33:46.583 INFO:tasks.workunit.client.0.smithi148.stderr:186236: 2024-02-05T05:33:42.188+0000 7f59d7924640 -1 Falling back to public interface 2024-02-05T05:33:46.583 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:46.583 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:46.583 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:46.583 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:33:46.583 INFO:tasks.workunit.client.0.smithi148.stderr:186236: 1 2024-02-05T05:33:46.584 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:46.584 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:33:46.584 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:46.584 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:46.584 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:46.584 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:33:46.584 INFO:tasks.workunit.client.0.smithi148.stderr:186236: 2 2024-02-05T05:33:46.584 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:46.584 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:33:46.584 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:46.584 INFO:tasks.workunit.client.0.smithi148.stderr:186236: 2024-02-05T05:33:45.368+0000 7f59d7924640 -1 osd.1 72 log_to_monitors true 2024-02-05T05:33:46.584 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:46.584 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:46.585 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:33:46.585 INFO:tasks.workunit.client.0.smithi148.stderr:186236: 3 2024-02-05T05:33:46.585 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:46.585 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:33:47.109 INFO:tasks.workunit.client.0.smithi148.stderr:186236: osd.1 up in weight 1 up_from 79 up_thru 70 down_at 73 last_clean_interval [70,72) [v2:127.0.0.1:6810/4293788619,v1:127.0.0.1:6811/4293788619] [v2:127.0.0.1:6812/4293788619,v1:127.0.0.1:6813/4293788619] exists,up e636a31a-b223-4902-beab-826a6cd4b5d7 2024-02-05T05:33:47.109 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:33:47.109 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:33:47.109 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:33:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:33:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:33:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:33:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:33:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:33:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:186238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:33:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:186238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:33:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:33:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:33:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:33:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:33:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:33:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:33:47.110 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:33:47.111 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:33:47.111 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:33:47.111 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:33:47.111 INFO:tasks.workunit.client.0.smithi148.stderr:186238: 1 2024-02-05T05:33:47.111 INFO:tasks.workunit.client.0.smithi148.stderr:186238: 2 2024-02-05T05:33:47.111 INFO:tasks.workunit.client.0.smithi148.stderr:186238: 3' 2024-02-05T05:33:47.111 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:33:47.111 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:47.111 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:33:47.111 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836540 2024-02-05T05:33:47.111 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836540 2024-02-05T05:33:47.111 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540' 2024-02-05T05:33:47.111 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:47.111 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:33:47.112 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416386 2024-02-05T05:33:47.112 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416386 2024-02-05T05:33:47.112 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-339302416386' 2024-02-05T05:33:47.112 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:47.638 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stat 73 last_clean_interval [69,72) [v2:127.0.0.1:6818/2023893338,v1:127.0.0.1:6819/2023893338] [v2:127.0.0.1:6820/2023893338,v1:127.0.0.1:6821/2023893338] exists,up adc22c43-3a38-4d23-933d-b1fb1e5ffc07 2024-02-05T05:33:47.638 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:33:47.638 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:33:47.638 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:33:47.638 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:33:47.638 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:33:47.638 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:33:47.638 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:33:47.638 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:33:47.638 INFO:tasks.workunit.client.0.smithi148.stderr:186236: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:33:47.638 INFO:tasks.workunit.client.0.smithi148.stderr:186236: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:33:47.639 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:33:47.639 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:33:47.639 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:33:47.639 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:33:47.639 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:33:47.639 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:33:47.639 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:33:47.639 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:33:47.639 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:33:47.639 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:33:47.639 INFO:tasks.workunit.client.0.smithi148.stderr:186236: 1 2024-02-05T05:33:47.639 INFO:tasks.workunit.client.0.smithi148.stderr:186236: 2 2024-02-05T05:33:47.639 INFO:tasks.workunit.client.0.smithi148.stderr:186236: 3' 2024-02-05T05:33:47.640 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:33:47.640 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:47.640 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:33:47.640 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836541 2024-02-05T05:33:47.640 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836541 2024-02-05T05:33:47.640 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541' 2024-02-05T05:33:47.640 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:47.640 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:33:47.640 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416387 2024-02-05T05:33:47.640 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416387 2024-02-05T05:33:47.640 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-339302416387' 2024-02-05T05:33:47.640 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:49.097 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:33:49.097 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481795 2024-02-05T05:33:49.097 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481795 2024-02-05T05:33:49.097 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-339302416387 2-330712481795' 2024-02-05T05:33:49.097 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:49.097 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:33:49.097 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972365 2024-02-05T05:33:49.097 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972365 2024-02-05T05:33:49.097 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-339302416387 2-330712481795 3-266287972365' 2024-02-05T05:33:49.098 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:49.098 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836541 2024-02-05T05:33:49.098 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:49.098 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:33:49.098 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836541 2024-02-05T05:33:49.098 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:49.098 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836541 2024-02-05T05:33:49.098 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-02-05T05:33:49.098 INFO:tasks.workunit.client.0.smithi148.stderr:186236: waiting osd.0 seq 21474836541 2024-02-05T05:33:49.098 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:49.098 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836541 2024-02-05T05:33:49.098 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:49.098 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-339302416387 2024-02-05T05:33:49.098 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:49.099 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:33:49.099 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-339302416387 2024-02-05T05:33:49.099 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:49.099 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416387 2024-02-05T05:33:49.099 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 339302416387' 2024-02-05T05:33:49.099 INFO:tasks.workunit.client.0.smithi148.stderr:186236: waiting osd.1 seq 339302416387 2024-02-05T05:33:49.099 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:33:49.099 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416387 -lt 339302416387 2024-02-05T05:33:49.099 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:49.099 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481795 2024-02-05T05:33:49.099 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:49.099 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:33:49.099 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481795 2024-02-05T05:33:49.774 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/uts: ceph tell osd.2 flush_pg_stats 2024-02-05T05:33:49.774 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-02-05T05:33:49.774 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-02-05T05:33:49.774 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-339302416386 2-330712481794' 2024-02-05T05:33:49.774 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:49.774 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:33:49.774 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972364 2024-02-05T05:33:49.775 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972364 2024-02-05T05:33:49.775 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-339302416386 2-330712481794 3-266287972364' 2024-02-05T05:33:49.775 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:49.775 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836540 2024-02-05T05:33:49.775 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:49.775 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:33:49.775 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836540 2024-02-05T05:33:49.775 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:49.775 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836540 2024-02-05T05:33:49.775 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-02-05T05:33:49.775 INFO:tasks.workunit.client.0.smithi148.stderr:186238: waiting osd.0 seq 21474836540 2024-02-05T05:33:49.775 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836540 2024-02-05T05:33:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:33:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:33:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836540 2024-02-05T05:33:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-339302416386 2024-02-05T05:33:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:33:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-339302416386 2024-02-05T05:33:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416386 2024-02-05T05:33:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 339302416386' 2024-02-05T05:33:49.777 INFO:tasks.workunit.client.0.smithi148.stderr:186238: waiting osd.1 seq 339302416386 2024-02-05T05:33:49.777 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:33:49.777 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416387 -lt 339302416386 2024-02-05T05:33:51.199 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:51.200 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481794 2024-02-05T05:33:51.200 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:51.200 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:33:51.200 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:51.200 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481794 2024-02-05T05:33:51.200 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-02-05T05:33:51.200 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481794' 2024-02-05T05:33:51.200 INFO:tasks.workunit.client.0.smithi148.stderr:186238: waiting osd.2 seq 330712481794 2024-02-05T05:33:51.200 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:33:51.200 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972364 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972364 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972364 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972364' 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: waiting osd.3 seq 266287972364 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972365 -lt 266287972364 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:33:51.202 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:33:51.202 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:33:51.202 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:33:51.202 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:33:51.202 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:33:51.202 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:33:51.202 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:33:51.202 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T05:33:51.202 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:33:51.834 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtebuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:51.835 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481795 2024-02-05T05:33:51.835 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481795' 2024-02-05T05:33:51.835 INFO:tasks.workunit.client.0.smithi148.stderr:186236: waiting osd.2 seq 330712481795 2024-02-05T05:33:51.835 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:33:51.835 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481794 -lt 330712481795 2024-02-05T05:33:51.835 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:33:51.835 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:33:51.835 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:33:51.835 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481795 2024-02-05T05:33:51.835 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:51.835 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:51.835 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972365 2024-02-05T05:33:51.836 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:33:51.836 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972365 2024-02-05T05:33:51.836 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:51.836 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972365 2024-02-05T05:33:51.836 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972365' 2024-02-05T05:33:51.836 INFO:tasks.workunit.client.0.smithi148.stderr:186236: waiting osd.3 seq 266287972365 2024-02-05T05:33:51.836 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:33:51.836 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972365 -lt 266287972365 2024-02-05T05:33:51.836 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:33:51.836 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:51.836 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:51.836 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:33:51.836 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:33:51.836 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:33:51.837 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:33:51.837 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:33:51.837 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:33:51.837 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:33:51.837 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:33:51.837 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T05:33:51.837 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:33:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephst/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:52.368 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-02-05T05:33:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-02-05T05:33:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:33:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-02-05T05:33:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:33:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:33:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:33:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:33:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:33:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:33:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:33:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:33:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:33:52.369 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:33:52.370 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:33:52.370 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:33:52.370 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:52.370 INFO:tasks.workunit.client.0.smithi148.stderr:186238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:52.370 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:33:52.370 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:33:52.370 INFO:tasks.workunit.client.0.smithi148.stderr:186238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:33:52.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:33:52.989 INFO:tasks.workunit.client.0.smithi148.stderr:test/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:52.989 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:52.989 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi148.stderr:186236: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi148.stderr:186236: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 186235 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:33:52.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:33:52.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:33:53.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:33:53.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-02-05T05:33:53.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:33:53.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 189010"' 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 189010' 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 189011"' 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 189011' 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:33:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 189010 189011' 2024-02-05T05:33:53.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:33:53.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:33:53.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:33:53.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 189010 2024-02-05T05:33:53.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/189013: /' 2024-02-05T05:33:53.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:33:53.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/189015: /' 2024-02-05T05:33:56.102 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:33:56.102 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:33:56.102 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:33:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:33:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:33:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:33:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:33:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:33:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:33:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:33:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:33:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:33:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:33:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:33:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:33:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:33:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:33:56.104 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:33:56.104 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:33:56.104 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:33:56.104 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:33:56.104 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:33:56.104 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:33:56.104 INFO:tasks.workunit.client.0.smithi148.stderr:189015: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:33:56.104 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:33:56.104 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:33:56.104 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:33:56.104 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:33:56.104 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:33:56.104 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:33:56.104 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:33:56.105 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:33:56.108 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:33:56.108 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:33:56.108 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:33:56.108 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:33:56.108 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:33:56.108 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:33:56.108 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:33:56.108 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:33:56.108 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:33:56.108 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:33:56.108 INFO:tasks.workunit.client.0.smithi148.stderr:189015: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:33:56.109 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:33:56.110 INFO:tasks.workunit.client.0.smithi148.stderr:189015: start osd.1 2024-02-05T05:33:56.110 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:33:56.597 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:33:56.597 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:33:56.597 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:33:56.597 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:33:56.597 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:33:56.597 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:33:56.597 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:33:56.597 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:33:56.597 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:33:56.597 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:33:56.597 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:33:56.598 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:33:56.598 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:33:56.598 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:33:56.598 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:33:56.598 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:33:56.598 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:33:56.598 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:33:56.598 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:33:56.598 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:33:56.598 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:33:56.598 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:33:56.598 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:33:56.598 INFO:tasks.workunit.client.0.smithi148.stderr:189013: remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:33:56.598 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:33:56.599 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:33:56.599 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:33:56.599 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:33:56.599 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:33:56.599 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:33:56.599 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:33:56.599 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:33:56.600 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:33:56.600 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:33:56.600 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:33:56.601 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:33:56.601 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:33:56.601 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:33:56.601 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:33:56.601 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:33:56.601 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:33:56.601 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:33:56.601 INFO:tasks.workunit.client.0.smithi148.stderr:189013: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:33:56.601 INFO:tasks.workunit.client.0.smithi148.stderr:189013: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:33:56.601 INFO:tasks.workunit.client.0.smithi148.stderr:189013: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:33:56.601 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:33:56.601 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:33:56.601 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:33:56.601 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:33:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:33:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:33:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:33:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:33:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:33:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:33:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:33:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:33:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:33:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:189013: start osd.3 2024-02-05T05:33:56.602 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:34:00.800 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:34:00.800 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:34:00.800 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:34:00.800 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:34:00.800 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:34:00.800 INFO:tasks.workunit.client.0.smithi148.stderr:189015: 2024-02-05T05:33:56.140+0000 7ff7be1c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:00.800 INFO:tasks.workunit.client.0.smithi148.stderr:189015: 2024-02-05T05:33:56.140+0000 7ff7be1c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:00.800 INFO:tasks.workunit.client.0.smithi148.stderr:189015: 2024-02-05T05:33:56.144+0000 7ff7be1c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:00.801 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:34:00.801 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:34:00.801 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:34:00.801 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:34:00.801 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:34:00.801 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:00.801 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:34:00.801 INFO:tasks.workunit.client.0.smithi148.stderr:189015: 0 2024-02-05T05:34:00.801 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:00.801 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:00.801 INFO:tasks.workunit.client.0.smithi148.stderr:189015: 2024-02-05T05:33:56.711+0000 7ff7be1c6640 -1 Falling back to public interface 2024-02-05T05:34:00.801 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:00.801 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: 1 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: 2 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: 2024-02-05T05:33:59.495+0000 7ff7be1c6640 -1 osd.1 82 log_to_monitors true 2024-02-05T05:34:00.802 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:00.803 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:00.803 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:34:00.803 INFO:tasks.workunit.client.0.smithi148.stderr:189015: 3 2024-02-05T05:34:00.803 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:00.803 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:01.452 INFO:tasks.workunit.client.0.smithi148.stderr:189015: osd.1 up in weight 1 up_from 87 up_thru 79 down_at 83 last_189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:34:01.452 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:34:01.452 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:34:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:34:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:34:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:189013: 2024-02-05T05:33:56.631+0000 7fdb11bff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:189013: 2024-02-05T05:33:56.667+0000 7fdb11bff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:189013: 2024-02-05T05:33:56.667+0000 7fdb11bff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:34:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:34:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:34:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:34:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:34:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:34:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:189013: 0 2024-02-05T05:34:01.454 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:01.454 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:34:01.454 INFO:tasks.workunit.client.0.smithi148.stderr:189013: 2024-02-05T05:33:57.207+0000 7fdb11bff640 -1 Falling back to public interface 2024-02-05T05:34:01.454 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:01.454 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:01.454 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:01.454 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:34:01.454 INFO:tasks.workunit.client.0.smithi148.stderr:189013: 1 2024-02-05T05:34:01.454 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:01.454 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:34:01.454 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:34:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:189013: 2 2024-02-05T05:34:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:34:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:189013: 2024-02-05T05:34:00.431+0000 7fdb11bff640 -1 osd.3 82 log_to_monitors true 2024-02-05T05:34:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:34:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:189013: 3 2024-02-05T05:34:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:01.456 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:34:02.084 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperclean_interval [79,82) [v2:127.0.0.1:6818/3032754203,v1:127.0.0.1:6819/3032754203] [v2:127.0.0.1:6820/3032754203,v1:127.0.0.1:6821/3032754203] exists,up adc22c43-3a38-4d23-933d-b1fb1e5ffc07 2024-02-05T05:34:02.084 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:34:02.084 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:34:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:34:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:34:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:34:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:34:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:34:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:34:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:189015: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:34:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:189015: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:34:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:34:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:34:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:34:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:34:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:34:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:34:02.086 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:34:02.086 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:34:02.086 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:34:02.086 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:34:02.086 INFO:tasks.workunit.client.0.smithi148.stderr:189015: 1 2024-02-05T05:34:02.086 INFO:tasks.workunit.client.0.smithi148.stderr:189015: 2 2024-02-05T05:34:02.086 INFO:tasks.workunit.client.0.smithi148.stderr:189015: 3' 2024-02-05T05:34:02.086 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:34:02.087 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:02.087 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:34:02.087 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836545 2024-02-05T05:34:02.087 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836545 2024-02-05T05:34:02.087 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545' 2024-02-05T05:34:02.087 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:02.087 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:34:02.087 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=373662154754 2024-02-05T05:34:02.087 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 373662154754 2024-02-05T05:34:02.087 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-373662154754' 2024-02-05T05:34:02.087 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:03.582 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph s.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:03.582 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:03.582 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:03.582 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:34:03.582 INFO:tasks.workunit.client.0.smithi148.stderr:189013: 4 2024-02-05T05:34:03.582 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:03.582 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:34:03.582 INFO:tasks.workunit.client.0.smithi148.stderr:189013: osd.3 up in weight 1 up_from 89 up_thru 89 down_at 83 last_clean_interval [62,82) [v2:127.0.0.1:6826/402143418,v1:127.0.0.1:6827/402143418] [v2:127.0.0.1:6828/402143418,v1:127.0.0.1:6829/402143418] exists,up 0282ec22-c7d3-4fbd-9dce-8fb6471fec13 2024-02-05T05:34:03.582 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:34:03.582 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:34:03.583 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:34:03.583 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:34:03.583 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:34:03.583 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:34:03.583 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:34:03.583 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:34:03.583 INFO:tasks.workunit.client.0.smithi148.stderr:189013: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:34:03.583 INFO:tasks.workunit.client.0.smithi148.stderr:189013: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:34:03.583 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:34:03.583 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:34:03.583 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:34:03.583 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:34:03.583 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:34:03.583 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:34:03.584 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:34:03.584 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:34:03.584 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:34:03.584 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:34:03.584 INFO:tasks.workunit.client.0.smithi148.stderr:189013: 1 2024-02-05T05:34:03.584 INFO:tasks.workunit.client.0.smithi148.stderr:189013: 2 2024-02-05T05:34:03.584 INFO:tasks.workunit.client.0.smithi148.stderr:189013: 3' 2024-02-05T05:34:03.584 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:34:03.584 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:03.584 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:34:03.584 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836546 2024-02-05T05:34:03.584 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836546 2024-02-05T05:34:03.584 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546' 2024-02-05T05:34:03.585 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:04.506 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperstell osd.2 flush_pg_stats 2024-02-05T05:34:04.506 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481799 2024-02-05T05:34:04.506 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481799 2024-02-05T05:34:04.507 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-373662154754 2-330712481799' 2024-02-05T05:34:04.507 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:04.507 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:34:04.507 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089346 2024-02-05T05:34:04.507 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089346 2024-02-05T05:34:04.507 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-373662154754 2-330712481799 3-382252089346' 2024-02-05T05:34:04.508 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:04.508 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836545 2024-02-05T05:34:04.508 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:04.508 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:34:04.508 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836545 2024-02-05T05:34:04.508 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:04.509 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836545 2024-02-05T05:34:04.509 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2024-02-05T05:34:04.509 INFO:tasks.workunit.client.0.smithi148.stderr:189015: waiting osd.0 seq 21474836545 2024-02-05T05:34:04.509 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:34:04.509 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836545 2024-02-05T05:34:04.509 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:04.510 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-373662154754 2024-02-05T05:34:04.510 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:04.510 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:34:04.510 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-373662154754 2024-02-05T05:34:04.510 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:04.510 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=373662154754 2024-02-05T05:34:04.511 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 373662154754' 2024-02-05T05:34:04.511 INFO:tasks.workunit.client.0.smithi148.stderr:189015: waiting osd.1 seq 373662154754 2024-02-05T05:34:04.511 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:34:04.511 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 373662154754 2024-02-05T05:34:04.511 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:34:04.511 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:34:04.512 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:34:04.512 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 373662154755 -lt 373662154754 2024-02-05T05:34:04.512 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:04.686 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //hom.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:34:04.686 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=373662154755 2024-02-05T05:34:04.686 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 373662154755 2024-02-05T05:34:04.686 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-373662154755' 2024-02-05T05:34:04.686 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:04.686 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:34:04.686 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481800 2024-02-05T05:34:04.686 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481800 2024-02-05T05:34:04.686 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-373662154755 2-330712481800' 2024-02-05T05:34:04.686 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:04.686 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:34:04.687 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089347 2024-02-05T05:34:04.687 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089347 2024-02-05T05:34:04.687 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-373662154755 2-330712481800 3-382252089347' 2024-02-05T05:34:04.687 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:04.687 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836546 2024-02-05T05:34:04.687 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:04.687 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:34:04.687 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836546 2024-02-05T05:34:04.687 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:04.687 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836546 2024-02-05T05:34:04.687 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-02-05T05:34:04.687 INFO:tasks.workunit.client.0.smithi148.stderr:189013: waiting osd.0 seq 21474836546 2024-02-05T05:34:04.687 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:34:04.687 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836546 -lt 21474836546 2024-02-05T05:34:04.688 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:04.688 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-373662154755 2024-02-05T05:34:04.688 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:04.688 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:34:04.688 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-373662154755 2024-02-05T05:34:04.688 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:04.688 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=373662154755 2024-02-05T05:34:04.688 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 373662154755' 2024-02-05T05:34:04.688 INFO:tasks.workunit.client.0.smithi148.stderr:189013: waiting osd.1 seq 373662154755 2024-02-05T05:34:04.688 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:34:05.851 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: fe/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481799 2024-02-05T05:34:05.851 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:05.851 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:34:05.851 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481799 2024-02-05T05:34:05.851 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:05.851 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481799 2024-02-05T05:34:05.851 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481799' 2024-02-05T05:34:05.851 INFO:tasks.workunit.client.0.smithi148.stderr:189015: waiting osd.2 seq 330712481799 2024-02-05T05:34:05.851 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:34:05.851 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481800 -lt 330712481799 2024-02-05T05:34:05.852 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:05.852 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-382252089346 2024-02-05T05:34:05.852 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:05.852 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:34:05.852 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-382252089346 2024-02-05T05:34:05.852 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:05.852 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089346 2024-02-05T05:34:05.852 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 382252089346' 2024-02-05T05:34:05.852 INFO:tasks.workunit.client.0.smithi148.stderr:189015: waiting osd.3 seq 382252089346 2024-02-05T05:34:05.852 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:34:05.852 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089346 -lt 382252089346 2024-02-05T05:34:05.852 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:34:05.853 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:34:05.853 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:34:05.853 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:34:05.853 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:34:05.853 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:34:05.853 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:34:05.853 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:34:05.853 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:34:05.853 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:34:05.853 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:34:05.853 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:34:05.853 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:34:07.050 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:34:07.050 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:34:07.050 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-02-05T05:34:07.050 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-02-05T05:34:07.050 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:34:07.050 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T05:34:07.050 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:34:07.051 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:34:07.051 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:34:07.051 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:34:07.051 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:34:07.051 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:34:07.051 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:34:07.051 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:34:07.051 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:34:07.051 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:34:07.051 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:34:07.051 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:34:07.051 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:34:07.051 INFO:tasks.workunit.client.0.smithi148.stderr:189015: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:34:07.052 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:34:07.052 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:34:07.052 INFO:tasks.workunit.client.0.smithi148.stderr:189015: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:34:07.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:34:07.053 INFO:tasks.workunit.client.0.smithi148.stderr:lush_pg_stats: test 373662154755 -lt 373662154755 2024-02-05T05:34:07.053 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:07.053 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481800 2024-02-05T05:34:07.053 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:07.053 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:34:07.053 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481800 2024-02-05T05:34:07.053 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:07.053 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481800 2024-02-05T05:34:07.053 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481800' 2024-02-05T05:34:07.053 INFO:tasks.workunit.client.0.smithi148.stderr:189013: waiting osd.2 seq 330712481800 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481800 -lt 330712481800 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-382252089347 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-382252089347 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089347 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 382252089347' 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: waiting osd.3 seq 382252089347 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089346 -lt 382252089347 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:34:07.054 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:34:07.055 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:34:07.055 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089347 -lt 382252089347 2024-02-05T05:34:07.055 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:34:07.055 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:34:07.055 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:34:07.055 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:34:07.055 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:34:07.055 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:34:07.055 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:34:07.055 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:34:07.700 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:34:07.700 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:34:07.700 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:34:07.700 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:34:07.700 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:34:07.700 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:34:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:189013: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:34:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:34:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:34:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:189013: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:34:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:34:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:34:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 189011 2024-02-05T05:34:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:34:07.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:34:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:34:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:34:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:34:07.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:34:07.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:34:07.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:34:07.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:34:07.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:34:07.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:34:07.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:34:07.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:34:07.998 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:34:07.998 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:34:07.998 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:34:07.998 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:34:07.998 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:34:08.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:33:35.202321+0000 2024-02-05T05:34:08.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:34:08.593 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:34:08.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:33:35.202321+0000 2024-02-05T05:34:08.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:34:08.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:33:35.202321+0000 2024-02-05T05:34:08.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:34:08.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:34:08.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:34:08.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:34:08.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:34:08.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:34:08.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:34:08.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:34:08.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:33:35.202321+0000 '>' 2024-02-05T05:33:35.202321+0000 2024-02-05T05:34:08.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:34:09.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:34:09.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:34:09.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:34:09.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:34:09.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:34:09.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:34:09.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:34:10.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:33:35.202321+0000 '>' 2024-02-05T05:33:35.202321+0000 2024-02-05T05:34:10.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:34:11.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:34:11.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:34:11.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:34:11.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:34:11.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:34:11.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:34:11.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:34:11.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:33:35.202321+0000 '>' 2024-02-05T05:33:35.202321+0000 2024-02-05T05:34:11.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:34:12.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:34:12.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:34:12.537 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:34:12.537 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:34:12.537 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:34:12.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:34:12.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:34:12.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:34:09.526489+0000 '>' 2024-02-05T05:33:35.202321+0000 2024-02-05T05:34:12.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:34:12.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:34:12.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:34:12.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:34:12.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:34:12.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 191911"' 2024-02-05T05:34:12.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 191911' 2024-02-05T05:34:12.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:34:12.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:34:12.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:34:12.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 191912"' 2024-02-05T05:34:12.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 191912' 2024-02-05T05:34:12.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:34:12.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 191911 191912' 2024-02-05T05:34:12.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:34:12.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:34:12.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 191911 2024-02-05T05:34:12.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:34:12.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/191914: /' 2024-02-05T05:34:12.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:34:12.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/191916: /' 2024-02-05T05:34:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:34:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:34:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:34:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:34:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:34:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:34:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:34:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:34:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:34:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:34:15.903 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:34:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:34:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:34:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:34:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:34:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:34:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:34:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:34:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:34:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:34:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:34:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:34:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:34:15.904 INFO:tasks.workunit.client.0.smithi148.stderr:191914: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:34:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:191914: _ 2024-02-05T05:34:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:191914: hinfo_key 2024-02-05T05:34:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:191914: snapset 2024-02-05T05:34:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:34:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:34:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:34:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:34:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:34:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:34:15.905 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:34:15.907 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:34:15.907 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:34:15.907 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:34:15.907 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:34:15.907 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:34:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:34:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:34:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:34:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:34:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:34:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:34:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:191914: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:34:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:191914: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:191914: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:34:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:34:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:34:15.908 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:34:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:34:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:34:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:34:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:34:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:34:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:34:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:34:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:34:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:34:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:34:15.909 INFO:tasks.workunit.client.0.smithi148.stderr:191914: start osd.3 2024-02-05T05:34:16.458 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:34:16.458 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:34:16.459 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:34:16.460 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:34:16.460 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:34:16.460 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:34:16.460 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:34:16.460 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:34:16.460 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:34:16.460 INFO:tasks.workunit.client.0.smithi148.stderr:191916: _ 2024-02-05T05:34:16.460 INFO:tasks.workunit.client.0.smithi148.stderr:191916: hinfo_key 2024-02-05T05:34:16.460 INFO:tasks.workunit.client.0.smithi148.stderr:191916: snapset 2024-02-05T05:34:16.460 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:34:16.460 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:34:16.460 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:34:16.460 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:34:16.461 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:34:16.461 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:34:16.461 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:34:16.461 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:34:16.463 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:34:16.463 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:34:16.463 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:34:16.463 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:34:16.463 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:34:16.463 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:34:16.463 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:34:16.463 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:34:16.464 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:34:16.465 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:34:16.465 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:34:16.465 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:34:16.465 INFO:tasks.workunit.client.0.smithi148.stderr:191916: start osd.1 2024-02-05T05:34:20.204 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:34:20.204 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:34:20.204 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:34:20.204 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:34:20.204 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:34:20.204 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:34:20.204 INFO:tasks.workunit.client.0.smithi148.stderr:191914: 2024-02-05T05:34:15.939+0000 7fd304f46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:20.205 INFO:tasks.workunit.client.0.smithi148.stderr:191914: 2024-02-05T05:34:15.979+0000 7fd304f46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:20.205 INFO:tasks.workunit.client.0.smithi148.stderr:191914: 2024-02-05T05:34:15.983+0000 7fd304f46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:20.205 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:34:20.205 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:34:20.205 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:34:20.205 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:34:20.205 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:34:20.205 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:20.205 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:34:20.205 INFO:tasks.workunit.client.0.smithi148.stderr:191914: 0 2024-02-05T05:34:20.205 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:20.205 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:34:20.205 INFO:tasks.workunit.client.0.smithi148.stderr:191914: 2024-02-05T05:34:16.539+0000 7fd304f46640 -1 Falling back to public interface 2024-02-05T05:34:20.205 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:20.206 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:20.206 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:20.206 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:34:20.206 INFO:tasks.workunit.client.0.smithi148.stderr:191914: 1 2024-02-05T05:34:20.206 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:20.206 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:34:20.206 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:20.206 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:20.206 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:20.206 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:34:20.206 INFO:tasks.workunit.client.0.smithi148.stderr:191914: 2 2024-02-05T05:34:20.206 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:20.206 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:34:20.206 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:20.207 INFO:tasks.workunit.client.0.smithi148.stderr:191914: 2024-02-05T05:34:19.355+0000 7fd304f46640 -1 osd.3 90 log_to_monitors true 2024-02-05T05:34:20.207 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:20.207 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:20.207 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:34:20.207 INFO:tasks.workunit.client.0.smithi148.stderr:191914: 3 2024-02-05T05:34:20.207 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:21.223 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-02-05T05:34:21.224 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:34:21.224 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:34:21.224 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:34:21.224 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:34:21.224 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:34:21.224 INFO:tasks.workunit.client.0.smithi148.stderr:191916: 2024-02-05T05:34:16.491+0000 7f35c2f98640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:21.224 INFO:tasks.workunit.client.0.smithi148.stderr:191916: 2024-02-05T05:34:16.495+0000 7f35c2f98640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:21.224 INFO:tasks.workunit.client.0.smithi148.stderr:191916: 2024-02-05T05:34:16.495+0000 7f35c2f98640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:21.224 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:34:21.224 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:34:21.224 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:34:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:34:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:34:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:34:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:191916: 0 2024-02-05T05:34:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:191916: 2024-02-05T05:34:17.035+0000 7f35c2f98640 -1 Falling back to public interface 2024-02-05T05:34:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:34:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:191916: 1 2024-02-05T05:34:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:34:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:191916: 2 2024-02-05T05:34:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:191916: 2024-02-05T05:34:20.551+0000 7f35c2f98640 -1 osd.1 90 log_to_monitors true 2024-02-05T05:34:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:34:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:191916: 3 2024-02-05T05:34:21.227 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:21.227 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:21.388 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:34:21.388 INFO:tasks.workunit.client.0.smithi148.stderr:191914: osd.3 up in weight 1 up_from 95 up_thru 89 down_at 91 last_clean_interval [89,90) [v2:127.0.0.1:6818/393266083,v1:127.0.0.1:6819/393266083] [v2:127.0.0.1:6820/393266083,v1:127.0.0.1:6821/393266083] exists,up 0282ec22-c7d3-4fbd-9dce-8fb6471fec13 2024-02-05T05:34:21.388 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:34:21.388 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:34:21.388 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:34:21.388 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:34:21.388 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:34:21.388 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:34:21.389 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:34:21.389 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:34:21.389 INFO:tasks.workunit.client.0.smithi148.stderr:191914: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:34:21.389 INFO:tasks.workunit.client.0.smithi148.stderr:191914: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:34:21.389 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:34:21.389 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:34:21.389 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:34:21.389 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:34:21.389 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:34:21.389 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:34:21.389 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:34:21.389 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:34:21.390 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:34:21.390 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:34:21.390 INFO:tasks.workunit.client.0.smithi148.stderr:191914: 1 2024-02-05T05:34:21.390 INFO:tasks.workunit.client.0.smithi148.stderr:191914: 2 2024-02-05T05:34:21.390 INFO:tasks.workunit.client.0.smithi148.stderr:191914: 3' 2024-02-05T05:34:21.390 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:34:21.390 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:21.390 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:34:21.390 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836551 2024-02-05T05:34:21.390 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836551 2024-02-05T05:34:21.390 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551' 2024-02-05T05:34:21.390 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:21.391 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:34:21.391 INFO:tasks.workunit.client.0.smithi148.stderr:191914: Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T05:34:21.391 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-02-05T05:34:21.391 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-02-05T05:34:21.391 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-02-05T05:34:22.963 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:22.963 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:34:22.963 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481805 2024-02-05T05:34:22.963 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481805 2024-02-05T05:34:22.963 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 2-330712481805' 2024-02-05T05:34:22.963 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:22.963 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:34:22.963 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-02-05T05:34:22.964 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-02-05T05:34:22.964 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 2-330712481805 3-408021893122' 2024-02-05T05:34:22.964 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:22.964 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836551 2024-02-05T05:34:22.964 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:22.964 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:34:22.964 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836551 2024-02-05T05:34:22.964 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:22.964 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836551 2024-02-05T05:34:22.964 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836551' 2024-02-05T05:34:22.964 INFO:tasks.workunit.client.0.smithi148.stderr:191914: waiting osd.0 seq 21474836551 2024-02-05T05:34:22.964 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836551 -lt 21474836551 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481805 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481805 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481805 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481805' 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: waiting osd.2 seq 330712481805 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481805 -lt 330712481805 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-408021893122 2024-02-05T05:34:22.965 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:22.966 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:34:23.200 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/eph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:23.201 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:23.201 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:23.201 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:34:23.201 INFO:tasks.workunit.client.0.smithi148.stderr:191916: 4 2024-02-05T05:34:23.201 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:23.201 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:23.201 INFO:tasks.workunit.client.0.smithi148.stderr:191916: osd.1 up in weight 1 up_from 98 up_thru 87 down_at 91 last_clean_interval [87,90) [v2:127.0.0.1:6826/2628979016,v1:127.0.0.1:6827/2628979016] [v2:127.0.0.1:6828/2628979016,v1:127.0.0.1:6829/2628979016] exists,up adc22c43-3a38-4d23-933d-b1fb1e5ffc07 2024-02-05T05:34:23.201 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:34:23.201 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:34:23.201 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:34:23.201 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:34:23.202 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:34:23.203 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:34:23.203 INFO:tasks.workunit.client.0.smithi148.stderr:191916: 1 2024-02-05T05:34:23.203 INFO:tasks.workunit.client.0.smithi148.stderr:191916: 2 2024-02-05T05:34:23.203 INFO:tasks.workunit.client.0.smithi148.stderr:191916: 3' 2024-02-05T05:34:23.203 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:34:23.203 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:23.203 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:34:23.203 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836552 2024-02-05T05:34:23.203 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836552 2024-02-05T05:34:23.203 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552' 2024-02-05T05:34:23.203 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:25.316 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:34:25.317 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795010 2024-02-05T05:34:25.317 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795010 2024-02-05T05:34:25.317 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-420906795010' 2024-02-05T05:34:25.317 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:25.317 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:34:25.317 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481806 2024-02-05T05:34:25.317 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481806 2024-02-05T05:34:25.318 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-420906795010 2-330712481806' 2024-02-05T05:34:25.318 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:34:25.318 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:34:25.318 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893123 2024-02-05T05:34:25.318 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893123 2024-02-05T05:34:25.318 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-420906795010 2-330712481806 3-408021893123' 2024-02-05T05:34:25.318 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:25.319 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836552 2024-02-05T05:34:25.319 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:25.319 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:34:25.319 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836552 2024-02-05T05:34:25.319 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:25.319 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836552 2024-02-05T05:34:25.320 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836552' 2024-02-05T05:34:25.320 INFO:tasks.workunit.client.0.smithi148.stderr:191916: waiting osd.0 seq 21474836552 2024-02-05T05:34:25.320 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:34:25.320 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836551 -lt 21474836552 2024-02-05T05:34:25.320 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:34:25.320 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:34:25.320 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:34:25.321 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836552 -lt 21474836552 2024-02-05T05:34:25.321 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:25.321 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-420906795010 2024-02-05T05:34:25.321 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:25.321 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:34:25.321 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-420906795010 2024-02-05T05:34:25.321 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:26.049 INFO:tasks.workunit.client.0.smithi148.stderr:19191standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-408021893122 2024-02-05T05:34:26.049 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-02-05T05:34:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 408021893122' 2024-02-05T05:34:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:191914: waiting osd.3 seq 408021893122 2024-02-05T05:34:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:34:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 408021893122 2024-02-05T05:34:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:34:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:34:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:34:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893123 -lt 408021893122 2024-02-05T05:34:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:34:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:34:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:34:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:34:26.051 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:34:26.051 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:34:26.051 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:34:26.051 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:34:26.051 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:34:26.051 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:34:26.051 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:34:26.051 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:34:26.051 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:34:26.051 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:34:26.051 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:34:26.051 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:34:26.051 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T05:34:26.052 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:34:26.052 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:34:26.052 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:34:26.052 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:34:26.052 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:34:26.052 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:34:26.674 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795010 2024-02-05T05:34:26.674 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 420906795010' 2024-02-05T05:34:26.674 INFO:tasks.workunit.client.0.smithi148.stderr:191916: waiting osd.1 seq 420906795010 2024-02-05T05:34:26.674 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:34:26.674 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795010 -lt 420906795010 2024-02-05T05:34:26.674 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:26.674 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481806 2024-02-05T05:34:26.674 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:26.674 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:34:26.674 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481806 2024-02-05T05:34:26.674 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481806 2024-02-05T05:34:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481806' 2024-02-05T05:34:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:191916: waiting osd.2 seq 330712481806 2024-02-05T05:34:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:34:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481806 -lt 330712481806 2024-02-05T05:34:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:34:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-408021893123 2024-02-05T05:34:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:34:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:34:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-408021893123 2024-02-05T05:34:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:34:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893123 2024-02-05T05:34:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 408021893123' 2024-02-05T05:34:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:191916: waiting osd.3 seq 408021893123 2024-02-05T05:34:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:34:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893123 2024-02-05T05:34:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:34:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:34:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:34:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:34:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:34:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:34:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:34:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:34:26.729 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(7: wait_for_clean: get_num_active_clean 2024-02-05T05:34:26.729 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:34:26.729 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:34:26.729 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:34:26.729 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:34:26.729 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:34:26.729 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:34:26.729 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:34:26.729 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:34:26.730 INFO:tasks.workunit.client.0.smithi148.stderr:191914: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:34:26.730 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:34:26.730 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:34:26.730 INFO:tasks.workunit.client.0.smithi148.stderr:191914: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:34:26.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:34:26.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:34:26.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 191912 2024-02-05T05:34:27.331 INFO:tasks.workunit.client.0.smithi148.stderr:contains("stale") | not)' 2024-02-05T05:34:27.331 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:34:27.331 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:34:27.331 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:34:27.331 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:34:27.332 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:34:27.332 INFO:tasks.workunit.client.0.smithi148.stderr:191916: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:34:27.332 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:34:27.332 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:34:27.332 INFO:tasks.workunit.client.0.smithi148.stderr:191916: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:34:27.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:34:27.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:34:27.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:34:27.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:34:27.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:34:27.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:34:27.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:34:27.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:34:27.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:34:27.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:34:27.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:34:27.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:34:27.404 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:34:27.404 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:34:27.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:34:27.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:34:27.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:34:27.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:34:27.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:34:27.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:34:27.715 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:34:27.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:34:27.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:34:27.717 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:34:27.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:34:27.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:34:27.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:34:27.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:34:27.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:34:27.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:34:27.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:34:27.763 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:34:27.763 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:27.763 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:34:27.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:34:27.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:34:27.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:34:27.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:34:27.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:34:27.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:34:27.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:34:27.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:34:27.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:34:27.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:34:27.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:34:27.766 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:34:27.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:34:27.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:34:27.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:34:27.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:34:27.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:34:27.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:34:27.769 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:34:27.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:34:27.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:34:27.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:34:27.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:34:27.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:34:27.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:34:27.773 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:34:27.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:34:27.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:34:27.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:34:27.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:34:27.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:27.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:34:27.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:34:27.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:34:27.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:34:27.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:34:27.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:34:27.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:27.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:34:27.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T05:34:27.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:34:27.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:34:27.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:34:27.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:34:27.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-02-05T05:34:27.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-02-05T05:34:27.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-02-05T05:34:27.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-02-05T05:34:27.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-02-05T05:34:27.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-02-05T05:34:27.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:34:27.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:34:27.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:34:27.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:34:27.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:34:27.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:34:28.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:34:28.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:34:28.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:34:28.192 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:34:28.192 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:28.192 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:34:28.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:34:28.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:34:28.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:34:28.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:34:28.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:34:28.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:34:28.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:34:28.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:34:28.412 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:34:28.412 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:34:28.412 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:34:28.413 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:34:28.413 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:28.413 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:34:28.413 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:34:28.413 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:34:28.413 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T05:34:28.529 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:34:28.529 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:34:28.529 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:34:28.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:34:28.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:34:28.531 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:34:28.531 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:34:28.531 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:34:28.531 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:34:28.531 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:28.531 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:34:28.531 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:34:28.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:34:28.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T05:34:28.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-02-05T05:34:28.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:34:28.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:34:28.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:34:28.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:34:28.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:34:28.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:34:29.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:34:29.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:34:29.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:34:29.521 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:34:29.521 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:29.521 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:34:29.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:34:29.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:34:29.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:34:29.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-02-05T05:34:29.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:34:29.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-02-05T05:34:29.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:34:29.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:34:29.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:34:29.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:34:29.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:34:29.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:34:29.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:34:29.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:34:29.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:34:29.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:34:29.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:34:29.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:34:29.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:34:29.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:34:29.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:34:29.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:34:29.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:34:29.564 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:34:29.564 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:29.564 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:34:29.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:34:29.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:34:29.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:34:29.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:34:29.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:34:29.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:34:29.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:34:29.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:34:29.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:34:29.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:34:29.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:34:29.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:34:29.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:34:29.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:34:29.566 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:34:29.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cc2a69e6-d2de-4c27-a9c9-6a890da31749 2024-02-05T05:34:29.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cc2a69e6-d2de-4c27-a9c9-6a890da31749' 2024-02-05T05:34:29.567 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 cc2a69e6-d2de-4c27-a9c9-6a890da31749 2024-02-05T05:34:29.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:34:29.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlc8BlsETIIhAAXILyiVBg0wz3EExVAZttIA== 2024-02-05T05:34:29.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlc8BlsETIIhAAXILyiVBg0wz3EExVAZttIA=="}' 2024-02-05T05:34:29.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cc2a69e6-d2de-4c27-a9c9-6a890da31749 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:34:29.847 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:34:29.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:34:29.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlc8BlsETIIhAAXILyiVBg0wz3EExVAZttIA== --osd-uuid cc2a69e6-d2de-4c27-a9c9-6a890da31749 2024-02-05T05:34:29.896 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:34:29.895+0000 7fab3806f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:29.914 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:34:29.911+0000 7fab3806f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:29.916 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:34:29.915+0000 7fab3806f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:29.917 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:34:29.915+0000 7fab3806f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:34:29.917 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:34:29.915+0000 7fab3806f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:34:29.917 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:34:29.915+0000 7fab3806f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:34:30.015 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:34:30.011+0000 7fab3806f640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:34:36.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:34:36.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:34:36.327 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:34:36.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:34:36.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:34:36.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:34:36.718 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:34:36.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:34:36.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:34:36.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:34:36.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:34:36.750 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:34:36.747+0000 7fac5a768640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:36.781 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:34:36.779+0000 7fac5a768640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:36.782 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:34:36.779+0000 7fac5a768640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:34:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:34:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:34:37.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:34:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:34:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:34:37.051 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:34:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:37.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:37.335 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:34:37.331+0000 7fac5a768640 -1 Falling back to public interface 2024-02-05T05:34:37.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:38.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:38.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:38.378 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:34:38.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:34:38.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:38.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:38.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:39.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:39.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:39.691 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:34:39.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:34:39.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:39.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:39.949 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:34:39.947+0000 7fac5a768640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:34:39.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:40.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:40.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:40.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:34:40.998 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:34:40.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:40.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:41.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:42.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:42.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:42.309 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:34:42.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:34:42.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:42.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:42.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:43.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:43.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:43.743 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:34:43.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:34:43.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:43.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:44.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:45.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:45.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:45.059 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:34:45.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:34:45.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:45.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:45.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:46.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:46.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:46.370 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:34:46.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:34:46.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:46.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:46.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:47.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:47.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:47.677 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:34:47.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:34:47.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:47.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:47.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:48.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:48.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:48.983 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:34:48.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:34:48.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:48.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:49.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:50.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:50.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:50.295 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:34:50.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:34:50.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:50.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:50.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:51.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:51.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:51.597 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:34:51.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:34:51.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:51.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:51.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:52.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:52.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:52.962 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:34:52.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:34:52.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:52.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:53.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:54.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:54.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:54.278 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:34:54.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:34:54.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:54.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:55.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:56.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:56.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:56.035 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:34:56.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:34:56.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:56.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:56.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:57.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:57.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:57.653 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:34:57.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:34:57.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:57.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:58.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:59.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:59.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:59.132 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:34:59.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:34:59.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:59.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:59.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:00.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:00.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:00.533 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:35:00.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:35:00.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:00.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:35:01.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:02.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:02.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:02.153 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:35:02.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:35:02.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:02.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:35:02.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:03.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:03.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:03.872 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:35:03.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:35:03.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:03.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:35:04.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:05.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:05.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:05.453 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:35:05.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:35:05.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:05.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:35:05.762 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1475379821,v1:127.0.0.1:6803/1475379821] [v2:127.0.0.1:6804/1475379821,v1:127.0.0.1:6805/1475379821] exists,up cc2a69e6-d2de-4c27-a9c9-6a890da31749 2024-02-05T05:35:05.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:35:05.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:35:05.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:35:05.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:35:05.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-02-05T05:35:05.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:35:05.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:35:05.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:35:05.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:35:05.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:35:05.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:35:05.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:35:05.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:35:05.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:35:05.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:35:05.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:35:05.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:35:05.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:35:05.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:35:05.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:35:05.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:35:05.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:35:05.767 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:35:05.767 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:35:05.767 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:35:05.767 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:35:05.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:35:05.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:35:05.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:35:05.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:35:05.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:35:05.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:35:05.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:35:05.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:35:05.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:35:05.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:35:05.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:35:05.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:35:05.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:35:05.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:35:05.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=df2657ff-4e94-4faa-a40f-4feff3561b69 2024-02-05T05:35:05.769 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 df2657ff-4e94-4faa-a40f-4feff3561b69 2024-02-05T05:35:05.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 df2657ff-4e94-4faa-a40f-4feff3561b69' 2024-02-05T05:35:05.769 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:35:05.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCJc8BlWo8dLxAAKsrz0clBcKr+hobWQy+rnQ== 2024-02-05T05:35:05.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCJc8BlWo8dLxAAKsrz0clBcKr+hobWQy+rnQ=="}' 2024-02-05T05:35:05.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new df2657ff-4e94-4faa-a40f-4feff3561b69 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:35:06.214 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:35:06.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:35:06.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCJc8BlWo8dLxAAKsrz0clBcKr+hobWQy+rnQ== --osd-uuid df2657ff-4e94-4faa-a40f-4feff3561b69 2024-02-05T05:35:06.269 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:06.266+0000 7fcd98774640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:06.336 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:06.334+0000 7fcd98774640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:06.338 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:06.334+0000 7fcd98774640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:06.339 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:06.334+0000 7fcd98774640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:35:06.339 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:06.334+0000 7fcd98774640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:35:06.339 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:06.334+0000 7fcd98774640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:35:06.390 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:06.390+0000 7fcd98774640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:35:13.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:35:13.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:35:13.111 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:35:13.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:35:13.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:35:13.495 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:35:13.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:35:13.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:35:13.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:35:13.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:35:13.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:35:13.527 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:13.526+0000 7fbfcbc9d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:13.529 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:13.530+0000 7fbfcbc9d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:13.530 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:13.530+0000 7fbfcbc9d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:13.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:35:13.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:35:13.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:35:13.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:35:13.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:35:13.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:13.799 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:35:13.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:35:13.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:13.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:14.074 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:14.074+0000 7fbfcbc9d640 -1 Falling back to public interface 2024-02-05T05:35:14.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:15.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:15.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:15.142 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:35:15.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:35:15.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:15.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:15.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:16.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:16.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:16.447 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:35:16.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:35:16.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:16.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:16.738 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:16.738+0000 7fbfcbc9d640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:35:16.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:17.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:17.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:17.749 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:35:17.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:35:17.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:17.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:18.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:19.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:19.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:19.124 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:35:19.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:35:19.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:19.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:19.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:20.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:20.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:20.542 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:35:20.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:35:20.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:20.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:20.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:21.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:21.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:21.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:35:21.962 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:35:21.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:21.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:22.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:23.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:23.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:23.293 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:35:23.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:35:23.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:23.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:23.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:24.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:24.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:24.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:35:24.735 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:35:24.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:24.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:25.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:26.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:26.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:26.112 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:35:26.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:35:26.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:26.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:26.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:27.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:27.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:27.433 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:35:27.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:35:27.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:27.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:27.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:28.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:28.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:28.795 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:35:28.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:35:28.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:28.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:29.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:30.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:30.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:30.237 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:35:30.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:35:30.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:30.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:30.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:31.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:31.552 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:35:31.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:35:31.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:31.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:32.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:33.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:33.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:33.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:35:33.231 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:35:33.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:33.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:33.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:34.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:34.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:34.947 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:35:34.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:35:34.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:34.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:35.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:36.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:36.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:36.765 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:35:36.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:35:36.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:36.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:37.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:38.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:38.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:38.402 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:35:38.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:35:38.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:38.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:38.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:39.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:39.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:39.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:35:39.846 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:35:39.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:39.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:40.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:41.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:41.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:41.385 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:35:41.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:35:41.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:41.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:42.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:43.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:43.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:43.197 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:35:43.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:35:43.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:43.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:43.496 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/749906256,v1:127.0.0.1:6811/749906256] [v2:127.0.0.1:6812/749906256,v1:127.0.0.1:6813/749906256] exists,up df2657ff-4e94-4faa-a40f-4feff3561b69 2024-02-05T05:35:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:35:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:35:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:35:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:35:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-02-05T05:35:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:35:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:35:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:35:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:35:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:35:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:35:43.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:35:43.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:35:43.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:35:43.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:35:43.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:35:43.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:35:43.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:35:43.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:35:43.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:35:43.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:35:43.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:35:43.500 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:35:43.500 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:35:43.501 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:35:43.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:35:43.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:35:43.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:35:43.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:35:43.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:35:43.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:35:43.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:35:43.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:35:43.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:35:43.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:35:43.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:35:43.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:35:43.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:35:43.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:35:43.503 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:35:43.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1a02051d-5e2d-4cfe-9d6d-594af054cd7c 2024-02-05T05:35:43.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1a02051d-5e2d-4cfe-9d6d-594af054cd7c' 2024-02-05T05:35:43.504 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 1a02051d-5e2d-4cfe-9d6d-594af054cd7c 2024-02-05T05:35:43.504 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:35:43.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvc8BlWeYRHxAAvZmHSBoZ8vZXhYApHhGt+g== 2024-02-05T05:35:43.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvc8BlWeYRHxAAvZmHSBoZ8vZXhYApHhGt+g=="}' 2024-02-05T05:35:43.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1a02051d-5e2d-4cfe-9d6d-594af054cd7c -i td/osd-scrub-repair/2/new.json 2024-02-05T05:35:43.947 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:35:43.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:35:43.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvc8BlWeYRHxAAvZmHSBoZ8vZXhYApHhGt+g== --osd-uuid 1a02051d-5e2d-4cfe-9d6d-594af054cd7c 2024-02-05T05:35:44.002 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:44.001+0000 7f838417d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:44.050 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:44.049+0000 7f838417d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:44.051 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:44.053+0000 7f838417d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:44.052 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:44.053+0000 7f838417d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:35:44.052 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:44.053+0000 7f838417d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:35:44.052 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:44.053+0000 7f838417d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:35:44.146 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:44.145+0000 7f838417d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:35:51.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:35:51.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:35:51.415 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T05:35:51.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:35:51.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:35:51.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:35:51.790 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T05:35:51.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:35:51.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:35:51.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:35:51.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:35:51.821 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:51.821+0000 7ff15d662640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:51.824 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:51.825+0000 7ff15d662640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:51.825 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:51.825+0000 7ff15d662640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:52.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:35:52.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:35:52.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:35:52.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:35:52.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:35:52.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:52.085 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:35:52.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:35:52.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:52.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:52.382 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:52.381+0000 7ff15d662640 -1 Falling back to public interface 2024-02-05T05:35:52.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:53.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:53.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:53.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:35:53.402 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:35:53.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:53.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:53.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:54.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:54.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:54.707 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:35:54.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:35:54.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:54.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:55.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:55.181 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:35:55.181+0000 7ff15d662640 -1 osd.2 0 log_to_monitors true 2024-02-05T05:35:56.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:56.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:56.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:35:56.030 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:35:56.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:56.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:56.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:57.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:57.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:57.484 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:35:57.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:35:57.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:57.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:57.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:58.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:58.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:58.805 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:35:58.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:35:58.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:58.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:59.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:00.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:00.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:00.139 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:36:00.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:36:00.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:00.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:00.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:01.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:01.455 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:36:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:36:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:01.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:01.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:02.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:02.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:02.765 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:36:02.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:36:02.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:02.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:03.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:04.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:04.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:04.372 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:36:04.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:36:04.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:04.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:04.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:05.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:05.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:05.728 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:36:05.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:36:05.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:05.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:06.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:07.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:07.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:07.262 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:36:07.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:36:07.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:07.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:07.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:08.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:08.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:08.572 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:36:08.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:36:08.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:08.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:08.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:09.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:09.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:09.933 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:36:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:36:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:10.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:11.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:11.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:11.761 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:36:11.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:36:11.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:11.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:12.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:13.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:13.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:13.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:36:13.401 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:36:13.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:13.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:14.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:15.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:15.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:15.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:36:15.241 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:36:15.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:15.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:16.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:17.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:17.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:17.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:36:17.100 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:36:17.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:17.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:17.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:18.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:18.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:18.928 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:36:18.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:36:18.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:18.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:19.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:20.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:20.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:20.612 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:36:20.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:36:20.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:20.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:21.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:22.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:22.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:22.124 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:36:22.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:36:22.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:22.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:22.450 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2926351300,v1:127.0.0.1:6819/2926351300] [v2:127.0.0.1:6820/2926351300,v1:127.0.0.1:6821/2926351300] exists,up 1a02051d-5e2d-4cfe-9d6d-594af054cd7c 2024-02-05T05:36:22.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:36:22.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:36:22.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:36:22.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:36:22.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-02-05T05:36:22.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:36:22.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:36:22.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T05:36:22.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:36:22.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:36:22.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:36:22.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:36:22.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:36:22.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:36:22.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:36:22.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:36:22.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:36:22.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:36:22.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:36:22.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:36:22.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:36:22.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:36:22.455 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:36:22.455 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:36:22.455 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:36:22.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:36:22.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:36:22.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:36:22.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:36:22.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:36:22.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:36:22.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:36:22.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:36:22.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:36:22.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:36:22.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:36:22.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:36:22.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:36:22.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:36:22.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:36:22.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b0f217d6-0908-40e7-b61a-8971d53ffef2 2024-02-05T05:36:22.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b0f217d6-0908-40e7-b61a-8971d53ffef2' 2024-02-05T05:36:22.457 INFO:tasks.workunit.client.0.smithi148.stdout:add osd3 b0f217d6-0908-40e7-b61a-8971d53ffef2 2024-02-05T05:36:22.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:36:22.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWc8Bl39KGHBAAhYNYtj6f4Sno4XkCg+8Geg== 2024-02-05T05:36:22.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWc8Bl39KGHBAAhYNYtj6f4Sno4XkCg+8Geg=="}' 2024-02-05T05:36:22.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b0f217d6-0908-40e7-b61a-8971d53ffef2 -i td/osd-scrub-repair/3/new.json 2024-02-05T05:36:22.885 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:36:22.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-02-05T05:36:22.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWc8Bl39KGHBAAhYNYtj6f4Sno4XkCg+8Geg== --osd-uuid b0f217d6-0908-40e7-b61a-8971d53ffef2 2024-02-05T05:36:22.941 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:36:22.940+0000 7f44ff4a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:22.995 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:36:22.992+0000 7f44ff4a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:22.997 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:36:22.996+0000 7f44ff4a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:22.997 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:36:22.996+0000 7f44ff4a6640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:36:22.998 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:36:22.996+0000 7f44ff4a6640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:36:22.998 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:36:22.996+0000 7f44ff4a6640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:36:23.081 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:36:23.080+0000 7f44ff4a6640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-02-05T05:36:29.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-02-05T05:36:29.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:36:29.814 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd3 key to auth repository 2024-02-05T05:36:29.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T05:36:29.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:36:30.205 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.3 2024-02-05T05:36:30.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T05:36:30.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:36:30.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:36:30.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:36:30.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:36:30.238 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:36:30.236+0000 7fc093757640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:30.239 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:36:30.240+0000 7fc093757640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:30.241 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:36:30.240+0000 7fc093757640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:30.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T05:36:30.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:36:30.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:36:30.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:36:30.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:36:30.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:30.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:36:30.513 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:36:30.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:30.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:30.794 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:36:30.792+0000 7fc093757640 -1 Falling back to public interface 2024-02-05T05:36:30.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:31.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:31.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:31.851 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:36:31.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:36:31.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:31.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:32.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:33.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:33.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:33.171 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:36:33.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:36:33.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:33.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:33.397 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:36:33.396+0000 7fc093757640 -1 osd.3 0 log_to_monitors true 2024-02-05T05:36:33.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:34.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:34.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:34.509 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:36:34.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:36:34.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:34.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:35.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:36.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:36.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:36.050 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:36:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:36:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:36.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:37.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:37.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:37.453 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:36:37.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:36:37.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:37.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:37.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:38.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:38.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:38.766 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:36:38.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:36:38.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:38.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:39.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:40.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:40.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:40.124 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:36:40.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:36:40.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:40.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:40.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:41.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:41.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:41.572 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:36:41.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:36:41.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:41.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:41.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:42.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:42.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:42.878 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:36:42.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:36:42.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:42.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:43.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:44.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:44.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:44.263 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:36:44.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:36:44.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:44.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:44.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:45.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:45.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:45.628 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:36:45.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:36:45.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:45.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:45.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:46.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:46.945 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:36:46.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:46.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:36:46.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:46.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:47.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:48.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:48.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:48.296 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:36:48.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:36:48.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:48.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:48.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:49.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:49.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:49.910 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:36:49.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:36:49.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:49.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:50.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:51.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:51.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:51.849 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:36:51.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:36:51.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:51.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:52.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:53.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:53.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:53.639 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:36:53.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:36:53.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:53.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:54.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:55.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:55.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:55.336 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:36:55.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:36:55.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:55.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:56.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:57.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:57.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:57.050 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:36:57.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:36:57.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:57.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:57.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:58.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:58.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:58.423 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:36:58.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:36:58.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:58.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:36:58.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:59.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:59.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:59.991 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:36:59.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:36:59.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:59.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:37:00.316 INFO:tasks.workunit.client.0.smithi148.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/612842302,v1:127.0.0.1:6827/612842302] [v2:127.0.0.1:6828/612842302,v1:127.0.0.1:6829/612842302] exists,up b0f217d6-0908-40e7-b61a-8971d53ffef2 2024-02-05T05:37:00.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:37:00.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:37:00.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:37:00.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:37:00.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-02-05T05:37:00.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:37:00.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:37:00.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-02-05T05:37:00.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:37:00.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-02-05T05:37:00.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:37:00.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:37:00.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:37:00.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:37:00.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-02-05T05:37:00.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-02-05T05:37:00.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:37:00.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:37:00.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:37:00.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:37:00.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:37:00.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:37:00.320 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:37:00.320 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:37:00.321 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:37:00.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:37:00.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:37:00.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:37:00.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:37:00.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:37:00.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:37:00.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:37:00.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:37:00.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:37:00.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:37:00.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:37:00.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:37:00.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:37:00.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-02-05T05:37:00.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:37:00.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f2b20155-5d1a-4714-a0d7-7df51cb2484e 2024-02-05T05:37:00.322 INFO:tasks.workunit.client.0.smithi148.stdout:add osd4 f2b20155-5d1a-4714-a0d7-7df51cb2484e 2024-02-05T05:37:00.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 f2b20155-5d1a-4714-a0d7-7df51cb2484e' 2024-02-05T05:37:00.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:37:00.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8c8BlXWFRFBAAGwvNf8Nteu5Vj9GADEK8HQ== 2024-02-05T05:37:00.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8c8BlXWFRFBAAGwvNf8Nteu5Vj9GADEK8HQ=="}' 2024-02-05T05:37:00.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f2b20155-5d1a-4714-a0d7-7df51cb2484e -i td/osd-scrub-repair/4/new.json 2024-02-05T05:37:00.823 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:37:00.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-02-05T05:37:00.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8c8BlXWFRFBAAGwvNf8Nteu5Vj9GADEK8HQ== --osd-uuid f2b20155-5d1a-4714-a0d7-7df51cb2484e 2024-02-05T05:37:00.881 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:00.880+0000 7f3064962640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:00.936 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:00.936+0000 7f3064962640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:00.939 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:00.936+0000 7f3064962640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:00.939 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:00.936+0000 7f3064962640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-02-05T05:37:00.939 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:00.936+0000 7f3064962640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-02-05T05:37:00.939 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:00.936+0000 7f3064962640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-02-05T05:37:01.043 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:01.044+0000 7f3064962640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-02-05T05:37:08.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-02-05T05:37:08.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:37:08.086 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd4 key to auth repository 2024-02-05T05:37:08.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-02-05T05:37:08.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:37:08.460 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.4 2024-02-05T05:37:08.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-02-05T05:37:08.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:37:08.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:37:08.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:37:08.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:37:08.493 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:08.491+0000 7f98971d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:08.528 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:08.527+0000 7f98971d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:08.529 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:08.527+0000 7f98971d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:08.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-02-05T05:37:08.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:37:08.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-02-05T05:37:08.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:37:08.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:37:08.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:08.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:37:08.805 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:37:08.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:08.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:09.102 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:09.099+0000 7f98971d2640 -1 Falling back to public interface 2024-02-05T05:37:09.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:10.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:10.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:10.141 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:37:10.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:37:10.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:10.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:10.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:11.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:11.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:11.458 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:37:11.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:37:11.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:11.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:11.690 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:11.687+0000 7f98971d2640 -1 osd.4 0 log_to_monitors true 2024-02-05T05:37:11.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:12.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:12.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:12.823 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:37:12.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:37:12.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:12.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:13.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:14.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:14.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:14.260 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:37:14.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:37:14.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:14.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:14.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:15.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:15.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:15.640 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:37:15.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:37:15.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:15.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:15.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:16.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:16.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:16.952 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:37:16.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:37:16.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:16.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:17.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:18.280 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:37:18.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:37:18.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:18.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:18.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:19.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:19.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:19.646 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:37:19.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:37:19.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:19.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:19.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:20.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:20.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:20.968 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:37:20.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:37:20.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:20.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:21.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:22.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:22.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:22.403 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:37:22.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:37:22.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:22.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:22.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:23.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:23.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:23.823 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:37:23.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:37:23.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:23.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:24.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:25.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:25.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:25.180 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:37:25.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:37:25.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:25.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:25.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:26.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:26.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:26.706 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:37:26.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:37:26.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:26.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:28.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:28.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:28.628 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:37:28.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:37:28.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:28.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:29.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:30.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:30.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:30.223 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:37:30.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:37:30.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:30.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:30.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:31.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:31.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:31.541 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:37:31.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:37:31.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:31.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:32.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:33.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:33.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:33.223 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:37:33.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:37:33.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:33.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:33.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:34.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:34.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:34.704 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:37:34.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:37:34.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:34.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:35.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:36.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:36.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:36.331 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:37:36.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:37:36.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:36.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:37.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:38.200 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:37:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:37:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:37:38.559 INFO:tasks.workunit.client.0.smithi148.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1319805380,v1:127.0.0.1:6835/1319805380] [v2:127.0.0.1:6836/1319805380,v1:127.0.0.1:6837/1319805380] exists,up f2b20155-5d1a-4714-a0d7-7df51cb2484e 2024-02-05T05:37:38.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:37:38.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:37:38.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:37:38.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:37:38.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-02-05T05:37:38.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:37:38.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:37:38.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-02-05T05:37:38.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:37:38.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:37:38.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:37:38.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:37:38.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:37:38.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:37:38.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:37:38.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:37:38.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:37:38.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:37:38.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:37:38.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:37:38.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:37:38.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:37:38.563 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:37:38.563 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:37:38.563 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:37:38.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:37:38.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:37:38.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:37:38.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:37:38.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:37:38.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:37:38.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:37:38.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:37:38.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:37:38.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:37:38.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:37:38.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:37:38.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:37:38.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:37:38.565 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:37:38.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f12201fc-c63f-4591-a81c-5c4646aaab07 2024-02-05T05:37:38.565 INFO:tasks.workunit.client.0.smithi148.stdout:add osd5 f12201fc-c63f-4591-a81c-5c4646aaab07 2024-02-05T05:37:38.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 f12201fc-c63f-4591-a81c-5c4646aaab07' 2024-02-05T05:37:38.566 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:37:38.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAidMBlk78UIxAAP8rUnbK6jpTjFAnzcmTv6A== 2024-02-05T05:37:38.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAidMBlk78UIxAAP8rUnbK6jpTjFAnzcmTv6A=="}' 2024-02-05T05:37:38.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f12201fc-c63f-4591-a81c-5c4646aaab07 -i td/osd-scrub-repair/5/new.json 2024-02-05T05:37:39.022 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:37:39.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-02-05T05:37:39.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAidMBlk78UIxAAP8rUnbK6jpTjFAnzcmTv6A== --osd-uuid f12201fc-c63f-4591-a81c-5c4646aaab07 2024-02-05T05:37:39.078 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:39.075+0000 7f6bd4581640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:39.132 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:39.131+0000 7f6bd4581640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:39.134 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:39.131+0000 7f6bd4581640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:39.135 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:39.131+0000 7f6bd4581640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-02-05T05:37:39.135 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:39.131+0000 7f6bd4581640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-02-05T05:37:39.135 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:39.131+0000 7f6bd4581640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-02-05T05:37:39.216 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:39.215+0000 7f6bd4581640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-02-05T05:37:46.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-02-05T05:37:46.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:37:46.341 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd5 key to auth repository 2024-02-05T05:37:46.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-02-05T05:37:46.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:37:46.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-02-05T05:37:46.771 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.5 2024-02-05T05:37:46.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:37:46.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:37:46.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:37:46.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:37:46.803 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:46.799+0000 7f913a465640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:46.805 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:46.803+0000 7f913a465640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:46.806 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:46.803+0000 7f913a465640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:47.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-02-05T05:37:47.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:37:47.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:37:47.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:37:47.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:37:47.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:47.137 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:37:47.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:37:47.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:47.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:37:47.370 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:47.367+0000 7f913a465640 -1 Falling back to public interface 2024-02-05T05:37:47.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:48.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:48.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:48.463 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:37:48.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:37:48.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:48.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:37:48.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:49.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:49.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:49.767 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:37:49.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:37:49.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:49.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:37:49.983 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:37:49.982+0000 7f913a465640 -1 osd.5 0 log_to_monitors true 2024-02-05T05:37:50.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:51.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:51.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:51.093 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:37:51.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:37:51.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:51.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:37:51.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:52.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:52.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:52.437 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:37:52.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:37:52.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:52.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:37:52.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:53.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:53.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:53.925 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:37:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:37:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:37:54.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:55.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:55.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:55.245 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:37:55.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:37:55.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:55.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:37:55.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:56.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:56.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:56.569 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:37:56.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:37:56.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:56.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:37:56.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:57.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:57.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:57.885 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:37:57.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:37:57.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:57.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:37:58.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:59.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:59.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:59.206 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:37:59.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:37:59.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:59.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:37:59.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:00.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:00.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:00.518 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:38:00.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:38:00.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:00.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:38:00.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:01.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:01.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:01.906 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:38:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:38:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:38:02.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:03.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:03.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:03.221 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:38:03.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:38:03.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:03.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:38:03.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:04.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:04.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:04.543 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:38:04.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:38:04.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:04.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:38:05.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:06.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:06.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:06.504 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:38:06.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:38:06.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:06.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:38:07.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:08.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:08.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:08.036 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:38:08.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:38:08.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:08.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:38:08.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:09.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:09.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:09.509 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:38:09.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:38:09.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:09.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:38:10.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:11.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:11.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:11.161 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:38:11.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:38:11.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:11.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:38:11.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:12.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:12.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:12.549 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:38:12.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:38:12.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:12.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:38:13.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:14.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:14.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:14.034 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:38:14.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:38:14.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:14.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:38:14.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:15.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:15.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:15.827 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:38:15.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:38:15.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:15.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:38:16.175 INFO:tasks.workunit.client.0.smithi148.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/682785392,v1:127.0.0.1:6843/682785392] [v2:127.0.0.1:6844/682785392,v1:127.0.0.1:6845/682785392] exists,up f12201fc-c63f-4591-a81c-5c4646aaab07 2024-02-05T05:38:16.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:38:16.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:38:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:38:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:38:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-02-05T05:38:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:38:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:38:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-02-05T05:38:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:38:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-02-05T05:38:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:38:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:38:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:38:16.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:38:16.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-02-05T05:38:16.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-02-05T05:38:16.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:38:16.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:38:16.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:38:16.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:38:16.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:38:16.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:38:16.179 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:38:16.179 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:38:16.179 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:38:16.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:38:16.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:38:16.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:38:16.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:38:16.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:38:16.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:38:16.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:38:16.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:38:16.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:38:16.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:38:16.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:38:16.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:38:16.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:38:16.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-02-05T05:38:16.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:38:16.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7e3e9cdc-12d4-447e-8d29-106b9fb888b6 2024-02-05T05:38:16.182 INFO:tasks.workunit.client.0.smithi148.stdout:add osd6 7e3e9cdc-12d4-447e-8d29-106b9fb888b6 2024-02-05T05:38:16.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 7e3e9cdc-12d4-447e-8d29-106b9fb888b6' 2024-02-05T05:38:16.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:38:16.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIdMBlRR1DDBAAI8ygGuC3yzkV0G5jln6rtQ== 2024-02-05T05:38:16.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIdMBlRR1DDBAAI8ygGuC3yzkV0G5jln6rtQ=="}' 2024-02-05T05:38:16.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7e3e9cdc-12d4-447e-8d29-106b9fb888b6 -i td/osd-scrub-repair/6/new.json 2024-02-05T05:38:16.578 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:38:16.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-02-05T05:38:16.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIdMBlRR1DDBAAI8ygGuC3yzkV0G5jln6rtQ== --osd-uuid 7e3e9cdc-12d4-447e-8d29-106b9fb888b6 2024-02-05T05:38:16.803 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:16.634+0000 7f921dd1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:16.803 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:16.646+0000 7f921dd1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:16.803 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:16.650+0000 7f921dd1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:16.804 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:16.650+0000 7f921dd1f640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-02-05T05:38:16.804 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:16.650+0000 7f921dd1f640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-02-05T05:38:16.804 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:16.650+0000 7f921dd1f640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-02-05T05:38:16.804 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:16.782+0000 7f921dd1f640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-02-05T05:38:22.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-02-05T05:38:22.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:38:22.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-02-05T05:38:22.877 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd6 key to auth repository 2024-02-05T05:38:22.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:38:23.262 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.6 2024-02-05T05:38:23.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-02-05T05:38:23.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:38:23.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:38:23.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:38:23.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:38:23.294 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:23.294+0000 7fa6781f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:23.296 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:23.298+0000 7fa6781f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:23.297 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:23.298+0000 7fa6781f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:23.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-02-05T05:38:23.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:38:23.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-02-05T05:38:23.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:38:23.587 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:38:23.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:38:23.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:23.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:38:23.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:23.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:23.850 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:23.850+0000 7fa6781f5640 -1 Falling back to public interface 2024-02-05T05:38:23.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:24.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:24.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:24.932 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:38:24.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:38:24.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:24.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:25.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:26.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:26.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:26.234 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:38:26.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:38:26.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:26.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:26.465 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:26.465+0000 7fa6781f5640 -1 osd.6 0 log_to_monitors true 2024-02-05T05:38:26.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:27.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:27.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:27.567 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:38:27.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:38:27.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:27.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:27.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:28.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:28.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:28.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:38:28.885 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:38:28.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:28.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:29.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:30.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:30.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:30.206 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:38:30.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:38:30.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:30.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:30.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:31.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:31.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:31.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:38:31.547 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:38:31.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:31.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:31.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:32.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:32.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:32.865 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:38:32.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:38:32.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:32.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:33.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:34.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:34.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:34.248 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:38:34.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:38:34.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:34.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:34.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:35.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:35.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:35.568 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:38:35.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:38:35.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:35.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:35.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:36.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:36.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:36.893 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:38:36.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:38:36.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:36.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:37.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:38.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:38.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:38.248 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:38:38.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:38:38.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:38.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:38.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:39.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:39.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:39.719 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:38:39.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:38:39.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:39.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:40.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:41.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:41.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:41.039 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:38:41.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:38:41.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:41.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:41.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:42.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:42.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:42.553 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:38:42.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:38:42.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:42.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:43.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:44.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:44.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:44.055 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:38:44.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:38:44.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:44.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:44.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:45.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:45.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:45.989 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:38:45.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:38:45.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:45.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:46.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:47.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:47.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:47.318 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:38:47.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:38:47.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:47.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:48.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:49.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:49.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:49.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:38:49.112 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:38:49.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:49.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:49.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:50.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:38:50.524 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:38:50.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:50.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:51.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:52.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:52.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:52.365 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:38:52.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:38:52.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:52.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:52.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:53.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:53.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:53.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T05:38:53.713 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T05:38:53.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:53.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:38:54.047 INFO:tasks.workunit.client.0.smithi148.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3426307794,v1:127.0.0.1:6851/3426307794] [v2:127.0.0.1:6852/3426307794,v1:127.0.0.1:6853/3426307794] exists,up 7e3e9cdc-12d4-447e-8d29-106b9fb888b6 2024-02-05T05:38:54.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:38:54.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:38:54.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:38:54.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:38:54.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-02-05T05:38:54.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:38:54.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:38:54.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-02-05T05:38:54.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:38:54.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-02-05T05:38:54.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:38:54.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:38:54.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:38:54.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:38:54.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-02-05T05:38:54.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-02-05T05:38:54.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:38:54.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:38:54.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:38:54.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:38:54.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:38:54.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:38:54.052 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:38:54.052 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:38:54.052 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:38:54.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:38:54.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:38:54.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:38:54.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:38:54.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:38:54.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:38:54.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:38:54.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:38:54.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:38:54.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:38:54.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:38:54.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:38:54.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:38:54.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-02-05T05:38:54.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:38:54.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=28f59dc6-55e7-4d7a-97ef-bf06f73872cc 2024-02-05T05:38:54.054 INFO:tasks.workunit.client.0.smithi148.stdout:add osd7 28f59dc6-55e7-4d7a-97ef-bf06f73872cc 2024-02-05T05:38:54.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 28f59dc6-55e7-4d7a-97ef-bf06f73872cc' 2024-02-05T05:38:54.054 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:38:54.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBudMBlwxZLBBAACV74sSWFLFdOwBuhQ73u9w== 2024-02-05T05:38:54.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBudMBlwxZLBBAACV74sSWFLFdOwBuhQ73u9w=="}' 2024-02-05T05:38:54.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 28f59dc6-55e7-4d7a-97ef-bf06f73872cc -i td/osd-scrub-repair/7/new.json 2024-02-05T05:38:54.447 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:38:54.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-02-05T05:38:54.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBudMBlwxZLBBAACV74sSWFLFdOwBuhQ73u9w== --osd-uuid 28f59dc6-55e7-4d7a-97ef-bf06f73872cc 2024-02-05T05:38:54.502 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:54.501+0000 7fe065559640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:54.558 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:54.557+0000 7fe065559640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:54.560 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:54.561+0000 7fe065559640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:54.560 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:54.561+0000 7fe065559640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-02-05T05:38:54.560 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:54.561+0000 7fe065559640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-02-05T05:38:54.560 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:54.561+0000 7fe065559640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-02-05T05:38:54.641 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:38:54.641+0000 7fe065559640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-02-05T05:39:01.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-02-05T05:39:01.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:39:01.300 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd7 key to auth repository 2024-02-05T05:39:01.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-02-05T05:39:01.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:39:01.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-02-05T05:39:01.685 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.7 2024-02-05T05:39:01.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:39:01.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:39:01.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:39:01.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:39:01.717 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:01.717+0000 7fce11e28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:39:01.751 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:01.749+0000 7fce11e28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:39:01.753 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:01.753+0000 7fce11e28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:39:02.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-02-05T05:39:02.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:39:02.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-02-05T05:39:02.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:39:02.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:39:02.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:02.024 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:39:02.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:39:02.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:02.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:02.297 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:02.297+0000 7fce11e28640 -1 Falling back to public interface 2024-02-05T05:39:02.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:03.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:03.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:03.379 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:39:03.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:39:03.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:03.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:03.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:04.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:04.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:04.695 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:39:04.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:39:04.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:04.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:04.980 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:04.980+0000 7fce11e28640 -1 osd.7 0 log_to_monitors true 2024-02-05T05:39:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:06.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:06.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:06.097 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:39:06.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:39:06.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:06.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:06.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:07.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:07.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:07.530 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:39:07.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:39:07.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:07.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:07.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:08.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:08.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:08.937 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:39:08.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:39:08.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:08.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:09.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:10.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:10.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:10.398 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:39:10.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:39:10.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:10.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:10.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:11.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:11.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:11.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:39:11.718 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:39:11.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:11.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:12.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:13.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:13.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:13.084 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:39:13.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:39:13.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:13.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:13.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:14.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:14.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:14.397 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:39:14.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:39:14.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:14.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:14.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:15.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:15.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:15.707 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:39:15.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:39:15.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:15.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:16.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:17.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:17.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:17.065 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:39:17.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:39:17.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:17.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:17.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:18.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:18.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:18.385 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:39:18.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:39:18.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:18.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:18.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:19.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:19.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:19.752 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:39:19.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:39:19.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:19.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:20.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:21.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:21.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:21.206 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:39:21.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:39:21.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:21.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:21.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:22.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:22.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:22.678 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:39:22.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:39:22.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:22.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:23.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:24.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:24.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:24.196 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:39:24.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:39:24.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:24.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:24.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:25.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:25.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:25.597 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:39:25.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:39:25.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:25.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:26.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:27.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:27.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:27.208 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:39:27.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:39:27.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:27.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:27.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:28.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:28.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:28.748 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:39:28.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:39:28.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:28.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:29.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:30.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:30.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:30.446 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:39:30.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:39:30.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:30.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:39:30.808 INFO:tasks.workunit.client.0.smithi148.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/980132271,v1:127.0.0.1:6859/980132271] [v2:127.0.0.1:6860/980132271,v1:127.0.0.1:6861/980132271] exists,up 28f59dc6-55e7-4d7a-97ef-bf06f73872cc 2024-02-05T05:39:30.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:39:30.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:39:30.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:39:30.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:39:30.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-02-05T05:39:30.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:39:30.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:39:30.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-02-05T05:39:30.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:39:30.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-02-05T05:39:30.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:39:30.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:39:30.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:39:30.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:39:30.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-02-05T05:39:30.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-02-05T05:39:30.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:39:30.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:39:30.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:39:30.811 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:39:30.811 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:39:30.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:39:30.812 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:39:30.812 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:39:30.812 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:39:30.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:39:30.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:39:30.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:39:30.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:39:30.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:39:30.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:39:30.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:39:30.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:39:30.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:39:30.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:39:30.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:39:30.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:39:30.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:39:30.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-02-05T05:39:30.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:39:30.815 INFO:tasks.workunit.client.0.smithi148.stdout:add osd8 5f835f0d-8454-484f-801f-c602674e785e 2024-02-05T05:39:30.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5f835f0d-8454-484f-801f-c602674e785e 2024-02-05T05:39:30.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 5f835f0d-8454-484f-801f-c602674e785e' 2024-02-05T05:39:30.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:39:30.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCSdMBltFXuMRAACPfIFuDSbCZ6eYxvcCJEsQ== 2024-02-05T05:39:30.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCSdMBltFXuMRAACPfIFuDSbCZ6eYxvcCJEsQ=="}' 2024-02-05T05:39:30.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5f835f0d-8454-484f-801f-c602674e785e -i td/osd-scrub-repair/8/new.json 2024-02-05T05:39:31.204 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:39:31.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-02-05T05:39:31.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCSdMBltFXuMRAACPfIFuDSbCZ6eYxvcCJEsQ== --osd-uuid 5f835f0d-8454-484f-801f-c602674e785e 2024-02-05T05:39:31.251 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:31.248+0000 7f8cc9d19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:39:31.297 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:31.296+0000 7f8cc9d19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:39:31.298 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:31.296+0000 7f8cc9d19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:39:31.299 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:31.296+0000 7f8cc9d19640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-02-05T05:39:31.299 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:31.296+0000 7f8cc9d19640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-02-05T05:39:31.299 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:31.296+0000 7f8cc9d19640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-02-05T05:39:31.382 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:31.380+0000 7f8cc9d19640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-02-05T05:39:38.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-02-05T05:39:38.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:39:38.165 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd8 key to auth repository 2024-02-05T05:39:38.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-02-05T05:39:38.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:39:38.549 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.8 2024-02-05T05:39:38.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-02-05T05:39:38.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:39:38.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:39:38.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:39:38.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:39:38.581 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:38.580+0000 7f3a4be04640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:39:38.584 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:38.584+0000 7f3a4be04640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:39:38.585 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:38.584+0000 7f3a4be04640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:39:38.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-02-05T05:39:38.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:39:38.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-02-05T05:39:38.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:39:38.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:39:38.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:38.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:39:38.890 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:39:38.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:38.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:39.158 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:39.156+0000 7f3a4be04640 -1 Falling back to public interface 2024-02-05T05:39:39.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:40.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:40.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:40.231 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:39:40.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:39:40.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:40.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:40.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:41.418 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:39:41.415+0000 7f3a4be04640 -1 osd.8 0 log_to_monitors true 2024-02-05T05:39:41.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:41.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:41.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:39:41.552 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:39:41.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:41.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:42.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:43.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:43.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:43.013 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:39:43.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:39:43.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:43.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:43.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:44.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:44.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:44.362 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:39:44.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:39:44.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:44.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:44.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:45.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:45.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:45.750 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:39:45.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:39:45.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:45.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:46.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:47.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:47.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:47.164 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:39:47.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:39:47.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:47.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:47.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:48.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:48.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:48.645 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:39:48.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:39:48.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:48.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:48.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:49.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:49.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:49.960 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:39:49.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:39:49.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:49.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:50.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:51.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:51.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:51.412 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:39:51.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:39:51.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:51.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:51.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:52.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:52.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:52.731 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:39:52.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:39:52.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:52.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:53.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:54.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:54.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:54.127 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:39:54.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:39:54.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:54.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:54.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:55.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:55.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:55.627 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:39:55.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:39:55.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:55.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:56.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:57.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:57.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:57.286 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:39:57.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:39:57.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:57.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:57.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:58.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:58.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:58.920 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:39:58.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:39:58.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:58.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:39:59.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:00.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:00.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:00.372 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:40:00.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:40:00.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:00.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:40:00.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:01.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:01.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:01.937 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:40:01.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:40:01.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:40:01.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:02.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:03.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:03.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:03.303 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:40:03.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:40:03.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:03.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:40:03.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:04.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:04.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:04.932 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:40:04.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:40:04.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:04.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:40:05.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:06.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:06.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:06.291 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:40:06.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:40:06.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:06.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:40:06.610 INFO:tasks.workunit.client.0.smithi148.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/1277760176,v1:127.0.0.1:6867/1277760176] [v2:127.0.0.1:6868/1277760176,v1:127.0.0.1:6869/1277760176] exists,up 5f835f0d-8454-484f-801f-c602674e785e 2024-02-05T05:40:06.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:40:06.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:40:06.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:40:06.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:40:06.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-02-05T05:40:06.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:40:06.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:40:06.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-02-05T05:40:06.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:40:06.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:40:06.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:40:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:40:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:40:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:40:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:40:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:40:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:40:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:40:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:40:06.613 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:40:06.613 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:40:06.613 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:40:06.616 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:40:06.616 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:40:06.616 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:40:06.616 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:40:06.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:40:06.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:40:06.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:40:06.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:40:06.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:40:06.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:40:06.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:40:06.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:40:06.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:40:06.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:40:06.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:40:06.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:40:06.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:40:06.617 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:40:06.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c49238e7-ce58-439e-bf47-f8337ce62570 2024-02-05T05:40:06.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 c49238e7-ce58-439e-bf47-f8337ce62570' 2024-02-05T05:40:06.618 INFO:tasks.workunit.client.0.smithi148.stdout:add osd9 c49238e7-ce58-439e-bf47-f8337ce62570 2024-02-05T05:40:06.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:40:06.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2dMBlVyQ2JhAA53n5PNqf6q5Zm7rDqgUgww== 2024-02-05T05:40:06.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2dMBlVyQ2JhAA53n5PNqf6q5Zm7rDqgUgww=="}' 2024-02-05T05:40:06.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c49238e7-ce58-439e-bf47-f8337ce62570 -i td/osd-scrub-repair/9/new.json 2024-02-05T05:40:07.021 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:40:07.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-02-05T05:40:07.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2dMBlVyQ2JhAA53n5PNqf6q5Zm7rDqgUgww== --osd-uuid c49238e7-ce58-439e-bf47-f8337ce62570 2024-02-05T05:40:07.077 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:40:07.075+0000 7f2272be3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:07.149 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:40:07.147+0000 7f2272be3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:07.150 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:40:07.147+0000 7f2272be3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:07.151 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:40:07.147+0000 7f2272be3640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-02-05T05:40:07.151 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:40:07.147+0000 7f2272be3640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-02-05T05:40:07.151 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:40:07.147+0000 7f2272be3640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-02-05T05:40:07.256 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:40:07.255+0000 7f2272be3640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-02-05T05:40:14.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-02-05T05:40:14.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:40:14.371 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd9 key to auth repository 2024-02-05T05:40:14.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-02-05T05:40:14.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:40:14.784 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.9 2024-02-05T05:40:14.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-02-05T05:40:14.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:40:14.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:40:14.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:40:14.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:40:14.816 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:40:14.815+0000 7fda896cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:14.818 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:40:14.815+0000 7fda896cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:14.819 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:40:14.819+0000 7fda896cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:15.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-02-05T05:40:15.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:40:15.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:40:15.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:40:15.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:40:15.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:15.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:40:15.117 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:40:15.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:15.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:15.366 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:40:15.363+0000 7fda896cb640 -1 Falling back to public interface 2024-02-05T05:40:15.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:16.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:16.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:16.438 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:40:16.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:40:16.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:16.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:16.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:17.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:17.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:17.782 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:40:17.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:40:17.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:17.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:17.959 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:40:17.955+0000 7fda896cb640 -1 osd.9 0 log_to_monitors true 2024-02-05T05:40:18.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:19.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:19.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:19.172 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:40:19.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:40:19.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:19.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:19.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:20.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:20.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:20.491 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:40:20.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:40:20.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:20.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:20.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:21.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:21.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:21.802 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:40:21.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:40:21.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:21.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:22.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:23.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:23.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:23.154 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:40:23.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:40:23.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:23.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:23.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:24.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:24.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:24.480 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:40:24.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:40:24.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:24.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:24.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:25.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:25.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:25.803 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:40:25.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:40:25.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:25.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:26.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:27.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:27.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:27.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:40:27.132 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:40:27.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:27.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:27.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:28.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:28.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:28.441 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:40:28.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:40:28.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:28.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:28.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:29.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:29.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:29.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:40:29.761 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:40:29.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:29.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:30.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:31.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:31.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:31.081 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:40:31.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:40:31.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:31.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:31.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:32.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:32.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:32.503 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:40:32.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:40:32.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:32.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:33.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:34.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:34.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:34.207 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:40:34.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:40:34.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:34.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:34.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:35.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:35.913 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:40:35.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:40:35.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:35.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:36.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:37.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:37.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:37.553 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:40:37.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:40:37.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:37.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:38.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:39.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:39.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:39.319 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:40:39.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:40:39.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:39.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:39.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:40.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:40.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:40.924 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:40:40.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:40:40.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:40.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:41.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:42.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:42.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:42.263 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:40:42.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:40:42.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:42.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:40:42.582 INFO:tasks.workunit.client.0.smithi148.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/4137512544,v1:127.0.0.1:6875/4137512544] [v2:127.0.0.1:6876/4137512544,v1:127.0.0.1:6877/4137512544] exists,up c49238e7-ce58-439e-bf47-f8337ce62570 2024-02-05T05:40:42.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:40:42.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:40:42.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:40:42.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-02-05T05:40:42.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:40:42.968 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T05:40:42.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:40:42.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:40:43.451 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T05:40:43.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:40:44.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:40:47.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-02-05T05:40:47.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:40:47.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:40:47.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:40:47.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:40:47.799 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:40:47.799 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:40:47.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:40:47.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:40:47.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:40:47.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:40:47.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:40:47.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:40:47.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:40:47.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:40:47.924 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:40:48.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:40:48.249 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:40:48.249 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:40:48.249 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:40:48.249 INFO:tasks.workunit.client.0.smithi148.stderr:4 2024-02-05T05:40:48.249 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:40:48.249 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:40:48.249 INFO:tasks.workunit.client.0.smithi148.stderr:7 2024-02-05T05:40:48.249 INFO:tasks.workunit.client.0.smithi148.stderr:8 2024-02-05T05:40:48.249 INFO:tasks.workunit.client.0.smithi148.stderr:9' 2024-02-05T05:40:48.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:40:48.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:48.250 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:40:48.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836553 2024-02-05T05:40:48.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836553 2024-02-05T05:40:48.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553' 2024-02-05T05:40:48.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:48.399 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:40:48.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673025 2024-02-05T05:40:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673025 2024-02-05T05:40:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025' 2024-02-05T05:40:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:48.542 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:40:48.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509498 2024-02-05T05:40:48.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509498 2024-02-05T05:40:48.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509498' 2024-02-05T05:40:48.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:48.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:40:48.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345974 2024-02-05T05:40:48.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345974 2024-02-05T05:40:48.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509498 3-85899345974' 2024-02-05T05:40:48.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:48.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:40:48.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182442 2024-02-05T05:40:48.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182442 2024-02-05T05:40:48.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509498 3-85899345974 4-107374182442' 2024-02-05T05:40:48.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:48.967 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:40:49.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018920 2024-02-05T05:40:49.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018920 2024-02-05T05:40:49.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509498 3-85899345974 4-107374182442 5-128849018920' 2024-02-05T05:40:49.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:49.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:40:49.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855387 2024-02-05T05:40:49.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855387 2024-02-05T05:40:49.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509498 3-85899345974 4-107374182442 5-128849018920 6-150323855387' 2024-02-05T05:40:49.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:49.258 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:40:49.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691861 2024-02-05T05:40:49.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691861 2024-02-05T05:40:49.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509498 3-85899345974 4-107374182442 5-128849018920 6-150323855387 7-171798691861' 2024-02-05T05:40:49.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:49.399 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:40:49.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528332 2024-02-05T05:40:49.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528332 2024-02-05T05:40:49.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509498 3-85899345974 4-107374182442 5-128849018920 6-150323855387 7-171798691861 8-193273528332' 2024-02-05T05:40:49.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:49.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:40:49.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364811 2024-02-05T05:40:49.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364811 2024-02-05T05:40:49.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673025 2-64424509498 3-85899345974 4-107374182442 5-128849018920 6-150323855387 7-171798691861 8-193273528332 9-214748364811' 2024-02-05T05:40:49.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:49.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836553 2024-02-05T05:40:49.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:49.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:40:49.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836553 2024-02-05T05:40:49.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:49.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836553 2024-02-05T05:40:49.690 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836553 2024-02-05T05:40:49.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-02-05T05:40:49.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:40:49.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836552 -lt 21474836553 2024-02-05T05:40:49.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:40:51.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:40:51.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:40:51.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836553 -lt 21474836553 2024-02-05T05:40:51.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:51.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673025 2024-02-05T05:40:51.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:51.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:40:51.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673025 2024-02-05T05:40:51.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:51.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673025 2024-02-05T05:40:51.325 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673025 2024-02-05T05:40:51.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673025' 2024-02-05T05:40:51.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:40:51.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673025 -lt 42949673025 2024-02-05T05:40:51.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:51.644 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509498 2024-02-05T05:40:51.644 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:51.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:40:51.647 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509498 2024-02-05T05:40:51.647 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:51.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509498 2024-02-05T05:40:51.649 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509498 2024-02-05T05:40:51.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2024-02-05T05:40:51.649 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:40:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509498 -lt 64424509498 2024-02-05T05:40:51.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:51.982 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:51.982 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345974 2024-02-05T05:40:51.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:40:51.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345974 2024-02-05T05:40:51.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:51.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345974 2024-02-05T05:40:51.986 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.3 seq 85899345974 2024-02-05T05:40:51.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345974' 2024-02-05T05:40:51.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:40:52.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345975 -lt 85899345974 2024-02-05T05:40:52.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:52.332 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182442 2024-02-05T05:40:52.332 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:52.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:40:52.334 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182442 2024-02-05T05:40:52.334 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:52.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182442 2024-02-05T05:40:52.335 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.4 seq 107374182442 2024-02-05T05:40:52.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182442' 2024-02-05T05:40:52.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:40:52.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182442 -lt 107374182442 2024-02-05T05:40:52.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:52.666 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018920 2024-02-05T05:40:52.667 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:52.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:40:52.669 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018920 2024-02-05T05:40:52.669 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:52.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018920 2024-02-05T05:40:52.672 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.5 seq 128849018920 2024-02-05T05:40:52.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018920' 2024-02-05T05:40:52.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:40:52.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018921 -lt 128849018920 2024-02-05T05:40:52.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:52.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855387 2024-02-05T05:40:52.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:52.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:40:52.992 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855387 2024-02-05T05:40:52.992 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:52.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855387 2024-02-05T05:40:52.993 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.6 seq 150323855387 2024-02-05T05:40:52.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855387' 2024-02-05T05:40:52.994 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:40:53.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855387 -lt 150323855387 2024-02-05T05:40:53.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:53.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691861 2024-02-05T05:40:53.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:53.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:40:53.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691861 2024-02-05T05:40:53.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:53.322 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.7 seq 171798691861 2024-02-05T05:40:53.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691861 2024-02-05T05:40:53.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691861' 2024-02-05T05:40:53.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:40:53.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691862 -lt 171798691861 2024-02-05T05:40:53.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:53.647 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528332 2024-02-05T05:40:53.647 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:53.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:40:53.649 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528332 2024-02-05T05:40:53.649 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:53.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528332 2024-02-05T05:40:53.651 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.8 seq 193273528332 2024-02-05T05:40:53.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528332' 2024-02-05T05:40:53.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:40:53.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528333 -lt 193273528332 2024-02-05T05:40:53.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:53.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:53.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364811 2024-02-05T05:40:53.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:40:53.973 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364811 2024-02-05T05:40:53.974 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:53.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364811 2024-02-05T05:40:53.976 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.9 seq 214748364811 2024-02-05T05:40:53.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364811' 2024-02-05T05:40:53.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:40:54.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364812 -lt 214748364811 2024-02-05T05:40:54.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:40:54.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:40:54.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:40:54.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:40:54.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:40:54.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:40:54.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:40:54.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:40:54.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:40:54.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:40:54.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:40:55.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:40:55.066 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:40:55.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:40:55.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:40:55.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:40:55.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:40:55.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:40:55.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-02-05T05:40:55.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:40:55.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:40:55.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-02-05T05:40:55.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:40:55.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-02-05T05:40:55.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:40:55.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:40:56.758 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'ecpool' created 2024-02-05T05:40:56.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:40:57.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-02-05T05:40:57.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:40:57.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:40:57.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:40:57.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:40:57.784 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:40:57.785 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:40:57.786 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:40:57.786 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:40:57.786 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:40:57.786 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:40:57.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:40:57.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:40:57.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:40:57.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:40:57.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:40:57.915 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:40:58.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:40:58.239 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:40:58.239 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:40:58.239 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:40:58.239 INFO:tasks.workunit.client.0.smithi148.stderr:4 2024-02-05T05:40:58.239 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:40:58.239 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:40:58.239 INFO:tasks.workunit.client.0.smithi148.stderr:7 2024-02-05T05:40:58.239 INFO:tasks.workunit.client.0.smithi148.stderr:8 2024-02-05T05:40:58.239 INFO:tasks.workunit.client.0.smithi148.stderr:9' 2024-02-05T05:40:58.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:40:58.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:58.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:40:58.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836556 2024-02-05T05:40:58.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836556 2024-02-05T05:40:58.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556' 2024-02-05T05:40:58.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:58.390 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:40:58.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673028 2024-02-05T05:40:58.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673028 2024-02-05T05:40:58.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673028' 2024-02-05T05:40:58.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:58.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:40:58.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509501 2024-02-05T05:40:58.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509501 2024-02-05T05:40:58.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673028 2-64424509501' 2024-02-05T05:40:58.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:58.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:40:59.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345977 2024-02-05T05:40:59.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345977 2024-02-05T05:40:59.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673028 2-64424509501 3-85899345977' 2024-02-05T05:40:59.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:59.134 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:40:59.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182445 2024-02-05T05:40:59.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182445 2024-02-05T05:40:59.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673028 2-64424509501 3-85899345977 4-107374182445' 2024-02-05T05:40:59.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:59.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:40:59.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018923 2024-02-05T05:40:59.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018923 2024-02-05T05:40:59.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673028 2-64424509501 3-85899345977 4-107374182445 5-128849018923' 2024-02-05T05:40:59.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:59.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:40:59.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855390 2024-02-05T05:40:59.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855390 2024-02-05T05:40:59.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673028 2-64424509501 3-85899345977 4-107374182445 5-128849018923 6-150323855390' 2024-02-05T05:40:59.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:59.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:40:59.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691864 2024-02-05T05:40:59.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691864 2024-02-05T05:40:59.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673028 2-64424509501 3-85899345977 4-107374182445 5-128849018923 6-150323855390 7-171798691864' 2024-02-05T05:40:59.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:59.828 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:40:59.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528335 2024-02-05T05:40:59.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528335 2024-02-05T05:40:59.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673028 2-64424509501 3-85899345977 4-107374182445 5-128849018923 6-150323855390 7-171798691864 8-193273528335' 2024-02-05T05:40:59.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:59.968 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:41:00.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364814 2024-02-05T05:41:00.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364814 2024-02-05T05:41:00.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673028 2-64424509501 3-85899345977 4-107374182445 5-128849018923 6-150323855390 7-171798691864 8-193273528335 9-214748364814' 2024-02-05T05:41:00.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:00.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836556 2024-02-05T05:41:00.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:00.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:41:00.113 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836556 2024-02-05T05:41:00.113 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:00.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836556 2024-02-05T05:41:00.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-02-05T05:41:00.115 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836556 2024-02-05T05:41:00.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:00.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836556 -lt 21474836556 2024-02-05T05:41:00.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:00.410 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673028 2024-02-05T05:41:00.410 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:00.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:41:00.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673028 2024-02-05T05:41:00.413 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:00.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673028 2024-02-05T05:41:00.414 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673028 2024-02-05T05:41:00.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673028' 2024-02-05T05:41:00.415 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:41:00.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673028 -lt 42949673028 2024-02-05T05:41:00.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:00.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:00.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509501 2024-02-05T05:41:00.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:41:00.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509501 2024-02-05T05:41:00.799 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:00.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509501 2024-02-05T05:41:00.800 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509501 2024-02-05T05:41:00.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509501' 2024-02-05T05:41:00.801 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:41:01.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509501 -lt 64424509501 2024-02-05T05:41:01.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:01.151 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345977 2024-02-05T05:41:01.151 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:01.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:41:01.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345977 2024-02-05T05:41:01.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:01.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345977 2024-02-05T05:41:01.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345977' 2024-02-05T05:41:01.154 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.3 seq 85899345977 2024-02-05T05:41:01.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:41:01.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345978 -lt 85899345977 2024-02-05T05:41:01.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:01.477 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182445 2024-02-05T05:41:01.477 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:01.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:41:01.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182445 2024-02-05T05:41:01.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:01.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182445 2024-02-05T05:41:01.481 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.4 seq 107374182445 2024-02-05T05:41:01.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182445' 2024-02-05T05:41:01.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:41:01.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182445 -lt 107374182445 2024-02-05T05:41:01.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:01.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018923 2024-02-05T05:41:01.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:01.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:41:01.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018923 2024-02-05T05:41:01.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:01.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018923 2024-02-05T05:41:01.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018923' 2024-02-05T05:41:01.802 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.5 seq 128849018923 2024-02-05T05:41:01.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:41:02.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018923 -lt 128849018923 2024-02-05T05:41:02.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:02.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855390 2024-02-05T05:41:02.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:02.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:41:02.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855390 2024-02-05T05:41:02.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:02.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855390 2024-02-05T05:41:02.158 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.6 seq 150323855390 2024-02-05T05:41:02.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855390' 2024-02-05T05:41:02.159 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:41:02.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855390 -lt 150323855390 2024-02-05T05:41:02.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:02.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691864 2024-02-05T05:41:02.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:02.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:41:02.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691864 2024-02-05T05:41:02.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:02.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691864 2024-02-05T05:41:02.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691864' 2024-02-05T05:41:02.488 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.7 seq 171798691864 2024-02-05T05:41:02.488 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:41:02.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691865 -lt 171798691864 2024-02-05T05:41:02.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:02.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528335 2024-02-05T05:41:02.804 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:02.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:41:02.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:02.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528335 2024-02-05T05:41:02.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528335 2024-02-05T05:41:02.807 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.8 seq 193273528335 2024-02-05T05:41:02.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528335' 2024-02-05T05:41:02.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:41:03.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528336 -lt 193273528335 2024-02-05T05:41:03.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:03.102 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364814 2024-02-05T05:41:03.102 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:03.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:41:03.105 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:03.105 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364814 2024-02-05T05:41:03.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364814 2024-02-05T05:41:03.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364814' 2024-02-05T05:41:03.106 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.9 seq 214748364814 2024-02-05T05:41:03.107 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:41:03.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364814 -lt 214748364814 2024-02-05T05:41:03.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:41:03.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:03.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:03.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:41:03.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:41:03.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:41:03.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:41:03.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:41:03.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:41:03.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:41:03.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:41:04.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:41:04.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:41:04.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:04.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:04.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:41:04.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:41:04.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:41:04.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:41:04.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-02-05T05:41:04.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T05:41:04.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-02-05T05:41:04.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-02-05T05:41:04.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:41:04.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:41:04.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T05:41:04.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:41:04.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:41:04.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:41:04.971 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T05:41:04.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:41:05.667 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T05:41:05.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:41:05.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:41:05.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:41:06.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-02-05T05:41:06.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T05:41:06.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:41:06.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:41:06.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:41:06.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-02-05T05:41:06.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-02-05T05:41:06.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T05:41:06.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-02-05T05:41:06.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-02-05T05:41:06.485 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:41:06.485 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:41:06.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-02-05T05:41:06.811 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:41:06.811 INFO:tasks.workunit.client.0.smithi148.stderr:9 2024-02-05T05:41:06.811 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:41:06.811 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:41:06.811 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:41:06.811 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:41:06.811 INFO:tasks.workunit.client.0.smithi148.stderr:7' 2024-02-05T05:41:06.811 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-02-05T05:41:06.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-02-05T05:41:06.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-02-05T05:41:06.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-02-05T05:41:06.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-02-05T05:41:06.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-02-05T05:41:06.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:41:06.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:41:06.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-02-05T05:41:06.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:41:06.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:41:07.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:41:07.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:41:07.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:41:07.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:41:07.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:41:07.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:41:07.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:41:07.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:41:09.384 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:41:09.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:41:09.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:41:09.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:41:09.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:41:09.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:41:09.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:41:09.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:41:09.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:41:09.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:41:09.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:41:09.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:41:09.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:41:09.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:41:09.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:41:09.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:41:09.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:41:09.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:41:09.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:41:09.933 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:41:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:41:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:41:09.934 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:41:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:41:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:41:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:41:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:41:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:41:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:41:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:41:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:41:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:41:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:41:09.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:41:09.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:41:09.935 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.3 2024-02-05T05:41:09.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:41:09.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:41:09.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:41:09.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:41:09.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:41:09.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:41:09.975 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:09.973+0000 7f85b556d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:09.978 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:09.977+0000 7f85b556d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:09.979 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:09.977+0000 7f85b556d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:10.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:41:10.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:41:10.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:41:10.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:41:10.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:41:10.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:10.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:41:10.258 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:41:10.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:10.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:41:10.514 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:10.513+0000 7f85b556d640 -1 Falling back to public interface 2024-02-05T05:41:10.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:11.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:11.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:11.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:41:11.557 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:41:11.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:11.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:41:11.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:12.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:12.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:12.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:41:12.874 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:41:12.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:12.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:41:13.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:13.201 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:13.201+0000 7f85b556d640 -1 osd.3 67 log_to_monitors true 2024-02-05T05:41:14.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:14.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:14.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:41:14.194 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:41:14.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:14.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:41:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:osd.3 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [20,67) [v2:127.0.0.1:6826/1205563443,v1:127.0.0.1:6827/1205563443] [v2:127.0.0.1:6828/1205563443,v1:127.0.0.1:6829/1205563443] exists,up b0f217d6-0908-40e7-b61a-8971d53ffef2 2024-02-05T05:41:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:41:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:41:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:41:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:41:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:41:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:41:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:41:14.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:41:14.749 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:41:14.749 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:41:14.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:41:14.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:41:14.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:41:14.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:41:14.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:41:14.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:41:14.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:41:14.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:41:14.876 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:41:15.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:41:15.268 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:41:15.268 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:41:15.268 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:41:15.268 INFO:tasks.workunit.client.0.smithi148.stderr:4 2024-02-05T05:41:15.268 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:41:15.268 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:41:15.268 INFO:tasks.workunit.client.0.smithi148.stderr:7 2024-02-05T05:41:15.268 INFO:tasks.workunit.client.0.smithi148.stderr:8 2024-02-05T05:41:15.268 INFO:tasks.workunit.client.0.smithi148.stderr:9' 2024-02-05T05:41:15.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:41:15.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:15.269 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:41:15.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836561 2024-02-05T05:41:15.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836561 2024-02-05T05:41:15.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561' 2024-02-05T05:41:15.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:15.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:41:15.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673032 2024-02-05T05:41:15.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673032 2024-02-05T05:41:15.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673032' 2024-02-05T05:41:15.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:15.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:41:15.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509505 2024-02-05T05:41:15.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509505 2024-02-05T05:41:15.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673032 2-64424509505' 2024-02-05T05:41:15.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:15.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:41:15.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678018 2024-02-05T05:41:15.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678018 2024-02-05T05:41:15.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673032 2-64424509505 3-304942678018' 2024-02-05T05:41:15.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:15.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:41:16.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182449 2024-02-05T05:41:16.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182449 2024-02-05T05:41:16.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673032 2-64424509505 3-304942678018 4-107374182449' 2024-02-05T05:41:16.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:16.029 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:41:16.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018927 2024-02-05T05:41:16.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018927 2024-02-05T05:41:16.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673032 2-64424509505 3-304942678018 4-107374182449 5-128849018927' 2024-02-05T05:41:16.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:16.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:41:16.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855394 2024-02-05T05:41:16.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855394 2024-02-05T05:41:16.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673032 2-64424509505 3-304942678018 4-107374182449 5-128849018927 6-150323855394' 2024-02-05T05:41:16.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:16.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:41:16.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691869 2024-02-05T05:41:16.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691869 2024-02-05T05:41:16.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673032 2-64424509505 3-304942678018 4-107374182449 5-128849018927 6-150323855394 7-171798691869' 2024-02-05T05:41:16.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:16.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:41:16.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528339 2024-02-05T05:41:16.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528339 2024-02-05T05:41:16.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673032 2-64424509505 3-304942678018 4-107374182449 5-128849018927 6-150323855394 7-171798691869 8-193273528339' 2024-02-05T05:41:16.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:16.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:41:16.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364818 2024-02-05T05:41:16.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364818 2024-02-05T05:41:16.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673032 2-64424509505 3-304942678018 4-107374182449 5-128849018927 6-150323855394 7-171798691869 8-193273528339 9-214748364818' 2024-02-05T05:41:16.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:16.905 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836561 2024-02-05T05:41:16.905 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:16.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:41:16.908 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836561 2024-02-05T05:41:16.908 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:16.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836561 2024-02-05T05:41:16.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2024-02-05T05:41:16.909 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836561 2024-02-05T05:41:16.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:17.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836561 -lt 21474836561 2024-02-05T05:41:17.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:17.216 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673032 2024-02-05T05:41:17.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:17.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:41:17.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673032 2024-02-05T05:41:17.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:17.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673032 2024-02-05T05:41:17.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673032' 2024-02-05T05:41:17.221 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949673032 2024-02-05T05:41:17.221 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:41:17.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673032 -lt 42949673032 2024-02-05T05:41:17.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:17.552 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509505 2024-02-05T05:41:17.552 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:17.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:41:17.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509505 2024-02-05T05:41:17.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:17.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509505 2024-02-05T05:41:17.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-02-05T05:41:17.556 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509505 2024-02-05T05:41:17.556 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:41:17.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509505 -lt 64424509505 2024-02-05T05:41:17.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:17.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678018 2024-02-05T05:41:17.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:17.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:41:17.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678018 2024-02-05T05:41:17.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:17.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678018 2024-02-05T05:41:17.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678018' 2024-02-05T05:41:17.874 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 304942678018 2024-02-05T05:41:17.875 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:41:18.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678018 -lt 304942678018 2024-02-05T05:41:18.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:18.193 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182449 2024-02-05T05:41:18.193 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:18.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:41:18.195 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182449 2024-02-05T05:41:18.195 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:18.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182449 2024-02-05T05:41:18.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182449' 2024-02-05T05:41:18.197 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.4 seq 107374182449 2024-02-05T05:41:18.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:41:18.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182449 -lt 107374182449 2024-02-05T05:41:18.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:18.507 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018927 2024-02-05T05:41:18.508 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:18.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:41:18.510 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018927 2024-02-05T05:41:18.510 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:18.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018927 2024-02-05T05:41:18.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018927' 2024-02-05T05:41:18.511 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.5 seq 128849018927 2024-02-05T05:41:18.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:41:18.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018928 -lt 128849018927 2024-02-05T05:41:18.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:18.840 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:18.840 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855394 2024-02-05T05:41:18.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:41:18.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855394 2024-02-05T05:41:18.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:18.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855394 2024-02-05T05:41:18.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855394' 2024-02-05T05:41:18.844 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.6 seq 150323855394 2024-02-05T05:41:18.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:41:19.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855394 -lt 150323855394 2024-02-05T05:41:19.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:19.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691869 2024-02-05T05:41:19.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:19.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:41:19.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691869 2024-02-05T05:41:19.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:19.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691869 2024-02-05T05:41:19.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691869' 2024-02-05T05:41:19.153 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.7 seq 171798691869 2024-02-05T05:41:19.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:41:19.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691869 -lt 171798691869 2024-02-05T05:41:19.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:19.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528339 2024-02-05T05:41:19.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:19.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:41:19.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528339 2024-02-05T05:41:19.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:19.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528339 2024-02-05T05:41:19.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528339' 2024-02-05T05:41:19.504 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.8 seq 193273528339 2024-02-05T05:41:19.504 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:41:19.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528340 -lt 193273528339 2024-02-05T05:41:19.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:19.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364818 2024-02-05T05:41:19.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:19.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:41:19.817 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364818 2024-02-05T05:41:19.817 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:19.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364818 2024-02-05T05:41:19.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364818' 2024-02-05T05:41:19.819 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.9 seq 214748364818 2024-02-05T05:41:19.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:41:20.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364819 -lt 214748364818 2024-02-05T05:41:20.151 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:41:20.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:20.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:20.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:41:20.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:41:20.537 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:41:20.537 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:41:20.537 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:41:20.537 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:41:20.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:41:20.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:41:20.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:41:20.852 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:41:20.852 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:20.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:21.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:41:21.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:41:21.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:41:21.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:41:21.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:41:21.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:41:21.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:41:21.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:41:21.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:41:21.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:41:21.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:41:21.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:41:21.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:41:21.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:41:21.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:41:21.565 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:41:21.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:40:56.483000+0000 2024-02-05T05:41:21.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:41:22.193 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:41:22.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:40:56.483000+0000 2024-02-05T05:41:22.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:41:22.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:40:56.483000+0000 2024-02-05T05:41:22.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:41:22.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:41:22.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:41:22.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:41:22.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:41:22.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:41:22.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:41:22.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:41:22.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:40:56.483000+0000 '>' 2024-02-05T05:40:56.483000+0000 2024-02-05T05:41:22.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:41:23.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:41:23.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:41:23.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:41:23.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:41:23.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:41:23.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:41:23.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:41:23.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:40:56.483000+0000 '>' 2024-02-05T05:40:56.483000+0000 2024-02-05T05:41:23.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:41:24.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:41:24.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:41:24.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:41:24.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:41:24.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:41:24.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:41:24.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:41:25.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:41:22.304157+0000 '>' 2024-02-05T05:40:56.483000+0000 2024-02-05T05:41:25.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:41:25.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:41:25.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:41:25.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:41:25.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:41:25.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:41:25.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:41:25.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:41:25.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:41:25.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:41:25.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:41:25.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:41:25.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:41:25.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:41:25.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:41:25.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:41:25.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:41:25.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:41:25.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:41:25.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:41:25.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:41:25.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:41:25.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:41:25.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:41:25.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-02-05T05:41:27.891 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:41:27.891 INFO:tasks.workunit.client.0.smithi148.stdout:hinfo_key 2024-02-05T05:41:27.891 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:41:28.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:41:28.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:41:28.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:41:28.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:41:28.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:41:28.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:41:28.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:41:28.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:41:28.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:41:28.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:41:28.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:41:28.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:41:28.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:41:28.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:41:28.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:41:28.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:41:28.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:41:28.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:41:28.433 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:41:28.433 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:41:28.433 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:41:28.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:41:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:41:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:41:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:41:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:41:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:41:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:41:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:41:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:41:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:41:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:41:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:41:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:41:28.436 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.9 2024-02-05T05:41:28.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:41:28.436 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:41:28.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:41:28.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:41:28.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:41:28.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:41:28.468 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:28.469+0000 7fec7436b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:28.471 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:28.469+0000 7fec7436b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:28.472 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:28.473+0000 7fec7436b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:28.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:41:28.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:41:28.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:41:28.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:41:28.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:41:28.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:28.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:41:28.752 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:41:28.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:28.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:41:29.005 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:29.005+0000 7fec7436b640 -1 Falling back to public interface 2024-02-05T05:41:29.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:30.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:30.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:30.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:41:30.080 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:41:30.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:30.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:41:30.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:31.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:31.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:31.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:41:31.384 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:41:31.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:31.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:41:31.538 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:31.537+0000 7fec7436b640 -1 osd.9 72 log_to_monitors true 2024-02-05T05:41:31.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:32.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:32.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:32.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:41:32.771 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:41:32.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:32.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:41:33.158 INFO:tasks.workunit.client.0.smithi148.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 73 last_clean_interval [50,72) [v2:127.0.0.1:6874/2344082120,v1:127.0.0.1:6875/2344082120] [v2:127.0.0.1:6876/2344082120,v1:127.0.0.1:6877/2344082120] exists,up c49238e7-ce58-439e-bf47-f8337ce62570 2024-02-05T05:41:33.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:41:33.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:41:33.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:41:33.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:41:33.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:41:33.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:41:33.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:41:33.159 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:41:33.159 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:41:33.159 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:41:33.160 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:41:33.160 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:41:33.160 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:41:33.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:41:33.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:41:33.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:41:33.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:41:33.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:41:33.258 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:41:33.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:41:33.583 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:41:33.583 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:41:33.583 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:41:33.583 INFO:tasks.workunit.client.0.smithi148.stderr:4 2024-02-05T05:41:33.583 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:41:33.583 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:41:33.583 INFO:tasks.workunit.client.0.smithi148.stderr:7 2024-02-05T05:41:33.583 INFO:tasks.workunit.client.0.smithi148.stderr:8 2024-02-05T05:41:33.583 INFO:tasks.workunit.client.0.smithi148.stderr:9' 2024-02-05T05:41:33.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:41:33.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:33.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:41:33.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836565 2024-02-05T05:41:33.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836565 2024-02-05T05:41:33.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565' 2024-02-05T05:41:33.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:33.893 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:41:34.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673037 2024-02-05T05:41:34.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673037 2024-02-05T05:41:34.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673037' 2024-02-05T05:41:34.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:34.044 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:41:34.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509510 2024-02-05T05:41:34.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509510 2024-02-05T05:41:34.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673037 2-64424509510' 2024-02-05T05:41:34.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:34.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:41:34.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678023 2024-02-05T05:41:34.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678023 2024-02-05T05:41:34.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673037 2-64424509510 3-304942678023' 2024-02-05T05:41:34.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:34.331 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:41:34.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182454 2024-02-05T05:41:34.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182454 2024-02-05T05:41:34.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673037 2-64424509510 3-304942678023 4-107374182454' 2024-02-05T05:41:34.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:34.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:41:34.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018932 2024-02-05T05:41:34.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018932 2024-02-05T05:41:34.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673037 2-64424509510 3-304942678023 4-107374182454 5-128849018932' 2024-02-05T05:41:34.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:34.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:41:34.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855399 2024-02-05T05:41:34.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855399 2024-02-05T05:41:34.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673037 2-64424509510 3-304942678023 4-107374182454 5-128849018932 6-150323855399' 2024-02-05T05:41:34.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:34.745 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:41:34.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691873 2024-02-05T05:41:34.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691873 2024-02-05T05:41:34.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673037 2-64424509510 3-304942678023 4-107374182454 5-128849018932 6-150323855399 7-171798691873' 2024-02-05T05:41:34.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:34.893 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:41:35.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528344 2024-02-05T05:41:35.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528344 2024-02-05T05:41:35.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673037 2-64424509510 3-304942678023 4-107374182454 5-128849018932 6-150323855399 7-171798691873 8-193273528344' 2024-02-05T05:41:35.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:35.037 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:41:35.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-02-05T05:41:35.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-02-05T05:41:35.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673037 2-64424509510 3-304942678023 4-107374182454 5-128849018932 6-150323855399 7-171798691873 8-193273528344 9-330712481794' 2024-02-05T05:41:35.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:35.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836565 2024-02-05T05:41:35.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:35.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:41:35.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836565 2024-02-05T05:41:35.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:35.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836565 2024-02-05T05:41:35.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-02-05T05:41:35.185 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836565 2024-02-05T05:41:35.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:35.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836565 -lt 21474836565 2024-02-05T05:41:35.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:35.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673037 2024-02-05T05:41:35.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:35.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:41:35.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673037 2024-02-05T05:41:35.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:35.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673037 2024-02-05T05:41:35.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673037' 2024-02-05T05:41:35.499 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949673037 2024-02-05T05:41:35.499 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:41:35.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673037 -lt 42949673037 2024-02-05T05:41:35.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:35.835 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:35.835 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509510 2024-02-05T05:41:35.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:41:35.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509510 2024-02-05T05:41:35.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:35.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509510 2024-02-05T05:41:35.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-02-05T05:41:35.838 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509510 2024-02-05T05:41:35.839 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:41:36.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509509 -lt 64424509510 2024-02-05T05:41:36.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:41:37.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:41:37.173 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:41:37.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509510 -lt 64424509510 2024-02-05T05:41:37.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:37.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678023 2024-02-05T05:41:37.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:37.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:41:37.488 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678023 2024-02-05T05:41:37.488 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:37.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678023 2024-02-05T05:41:37.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678023' 2024-02-05T05:41:37.490 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 304942678023 2024-02-05T05:41:37.490 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:41:37.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678023 -lt 304942678023 2024-02-05T05:41:37.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:37.804 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182454 2024-02-05T05:41:37.804 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:37.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:41:37.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182454 2024-02-05T05:41:37.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:37.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182454 2024-02-05T05:41:37.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182454' 2024-02-05T05:41:37.807 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.4 seq 107374182454 2024-02-05T05:41:37.808 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:41:38.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182454 -lt 107374182454 2024-02-05T05:41:38.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:38.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018932 2024-02-05T05:41:38.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:38.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:41:38.121 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018932 2024-02-05T05:41:38.121 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:38.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018932 2024-02-05T05:41:38.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018932' 2024-02-05T05:41:38.124 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.5 seq 128849018932 2024-02-05T05:41:38.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:41:38.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018933 -lt 128849018932 2024-02-05T05:41:38.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:38.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855399 2024-02-05T05:41:38.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:38.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:41:38.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855399 2024-02-05T05:41:38.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:38.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855399 2024-02-05T05:41:38.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855399' 2024-02-05T05:41:38.459 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.6 seq 150323855399 2024-02-05T05:41:38.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:41:38.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855399 -lt 150323855399 2024-02-05T05:41:38.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:38.785 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691873 2024-02-05T05:41:38.785 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:38.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:41:38.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691873 2024-02-05T05:41:38.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:38.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691873 2024-02-05T05:41:38.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2024-02-05T05:41:38.788 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.7 seq 171798691873 2024-02-05T05:41:38.789 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:41:39.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691874 -lt 171798691873 2024-02-05T05:41:39.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:39.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528344 2024-02-05T05:41:39.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:39.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:41:39.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528344 2024-02-05T05:41:39.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:39.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528344 2024-02-05T05:41:39.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528344' 2024-02-05T05:41:39.110 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.8 seq 193273528344 2024-02-05T05:41:39.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:41:39.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528345 -lt 193273528344 2024-02-05T05:41:39.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:39.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481794 2024-02-05T05:41:39.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:39.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:41:39.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481794 2024-02-05T05:41:39.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:39.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-02-05T05:41:39.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-02-05T05:41:39.471 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.9 seq 330712481794 2024-02-05T05:41:39.471 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:41:39.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-02-05T05:41:39.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:41:39.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:39.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:40.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:41:40.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:41:40.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:41:40.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:41:40.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:41:40.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:41:40.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:41:40.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:41:40.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:41:40.624 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:41:40.624 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:40.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:40.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:41:40.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:41:40.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:41:40.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:41:41.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:41:41.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-02-05T05:41:41.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:41:41.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:41:41.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-02-05T05:41:41.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:41:41.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:41:41.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:41:41.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:41:41.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:41:41.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:41:41.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:41:41.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:41:41.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:41:41.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:41:41.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:41:41.043 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:41:41.043 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:41:41.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:41:41.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:41:41.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:41:41.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:41:41.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:41:41.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:41:41.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:41:41.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:41:41.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:41:41.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:41:41.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-02-05T05:41:43.672 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:41:44.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:41:44.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:41:44.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:41:44.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:41:44.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:41:44.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:41:44.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:41:44.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:41:44.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:41:44.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:41:44.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:41:44.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:41:44.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:41:44.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:41:44.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:41:44.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:41:44.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:41:44.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:41:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:41:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:41:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:41:44.227 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:41:44.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:41:44.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:41:44.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:41:44.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:41:44.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:41:44.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:41:44.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:41:44.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:41:44.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:41:44.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:41:44.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:41:44.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:41:44.228 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.5 2024-02-05T05:41:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:41:44.229 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:41:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:41:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:41:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:41:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:41:44.267 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:44.264+0000 7f7d57dae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:44.300+0000 7f7d57dae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:44.300+0000 7f7d57dae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:44.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:41:44.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:41:44.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:41:44.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:41:44.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:41:44.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:44.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:41:44.557 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:41:44.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:44.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:41:44.842 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:44.840+0000 7f7d57dae640 -1 Falling back to public interface 2024-02-05T05:41:44.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:45.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:45.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:45.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:41:45.864 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:41:45.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:45.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:41:46.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:47.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:47.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:47.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:41:47.168 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:41:47.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:47.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:41:47.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:47.662 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:41:47.660+0000 7f7d57dae640 -1 osd.5 78 log_to_monitors true 2024-02-05T05:41:48.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:48.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:48.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:41:48.490 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:41:48.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:48.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:41:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:osd.5 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [30,78) [v2:127.0.0.1:6842/872436215,v1:127.0.0.1:6843/872436215] [v2:127.0.0.1:6844/872436215,v1:127.0.0.1:6845/872436215] exists,up f12201fc-c63f-4591-a81c-5c4646aaab07 2024-02-05T05:41:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:41:49.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:41:49.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:41:49.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:41:49.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:41:49.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:41:49.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:41:49.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:41:49.109 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:41:49.109 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:41:49.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:41:49.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:41:49.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:41:49.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:41:49.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:41:49.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:41:49.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:41:49.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:41:49.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:41:49.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:41:49.782 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:41:49.782 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:41:49.782 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:41:49.782 INFO:tasks.workunit.client.0.smithi148.stderr:4 2024-02-05T05:41:49.782 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:41:49.782 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:41:49.783 INFO:tasks.workunit.client.0.smithi148.stderr:7 2024-02-05T05:41:49.783 INFO:tasks.workunit.client.0.smithi148.stderr:8 2024-02-05T05:41:49.783 INFO:tasks.workunit.client.0.smithi148.stderr:9' 2024-02-05T05:41:49.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:41:49.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:49.783 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:41:50.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836569 2024-02-05T05:41:50.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836569 2024-02-05T05:41:50.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569' 2024-02-05T05:41:50.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:50.018 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:41:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673041 2024-02-05T05:41:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673041 2024-02-05T05:41:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673041' 2024-02-05T05:41:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:50.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:41:50.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509514 2024-02-05T05:41:50.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509514 2024-02-05T05:41:50.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673041 2-64424509514' 2024-02-05T05:41:50.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:50.409 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:41:50.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678027 2024-02-05T05:41:50.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678027 2024-02-05T05:41:50.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673041 2-64424509514 3-304942678027' 2024-02-05T05:41:50.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:50.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:41:50.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182458 2024-02-05T05:41:50.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182458 2024-02-05T05:41:50.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673041 2-64424509514 3-304942678027 4-107374182458' 2024-02-05T05:41:50.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:50.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:41:50.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-02-05T05:41:50.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-02-05T05:41:50.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673041 2-64424509514 3-304942678027 4-107374182458 5-352187318274' 2024-02-05T05:41:50.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:50.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:41:51.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855403 2024-02-05T05:41:51.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855403 2024-02-05T05:41:51.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673041 2-64424509514 3-304942678027 4-107374182458 5-352187318274 6-150323855403' 2024-02-05T05:41:51.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:51.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:41:51.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691878 2024-02-05T05:41:51.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691878 2024-02-05T05:41:51.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673041 2-64424509514 3-304942678027 4-107374182458 5-352187318274 6-150323855403 7-171798691878' 2024-02-05T05:41:51.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:51.169 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:41:51.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528348 2024-02-05T05:41:51.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528348 2024-02-05T05:41:51.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673041 2-64424509514 3-304942678027 4-107374182458 5-352187318274 6-150323855403 7-171798691878 8-193273528348' 2024-02-05T05:41:51.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:51.303 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:41:51.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-02-05T05:41:51.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-02-05T05:41:51.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673041 2-64424509514 3-304942678027 4-107374182458 5-352187318274 6-150323855403 7-171798691878 8-193273528348 9-330712481798' 2024-02-05T05:41:51.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:51.442 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836569 2024-02-05T05:41:51.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:51.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:41:51.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836569 2024-02-05T05:41:51.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:51.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836569 2024-02-05T05:41:51.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-02-05T05:41:51.445 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836569 2024-02-05T05:41:51.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:51.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836569 -lt 21474836569 2024-02-05T05:41:51.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:51.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673041 2024-02-05T05:41:51.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:51.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:41:51.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673041 2024-02-05T05:41:51.744 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:51.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673041 2024-02-05T05:41:51.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673041' 2024-02-05T05:41:51.745 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949673041 2024-02-05T05:41:51.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:41:52.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673040 -lt 42949673041 2024-02-05T05:41:52.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:41:53.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:41:53.068 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:41:53.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673041 -lt 42949673041 2024-02-05T05:41:53.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:53.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509514 2024-02-05T05:41:53.382 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:53.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:41:53.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509514 2024-02-05T05:41:53.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:53.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509514 2024-02-05T05:41:53.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-02-05T05:41:53.387 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509514 2024-02-05T05:41:53.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:41:53.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509515 -lt 64424509514 2024-02-05T05:41:53.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:53.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678027 2024-02-05T05:41:53.700 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:53.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:41:53.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678027 2024-02-05T05:41:53.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:53.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678027 2024-02-05T05:41:53.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678027' 2024-02-05T05:41:53.703 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 304942678027 2024-02-05T05:41:53.704 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:41:54.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678027 -lt 304942678027 2024-02-05T05:41:54.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:54.019 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182458 2024-02-05T05:41:54.019 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:54.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:41:54.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182458 2024-02-05T05:41:54.022 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:54.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182458 2024-02-05T05:41:54.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182458' 2024-02-05T05:41:54.023 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.4 seq 107374182458 2024-02-05T05:41:54.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:41:54.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182459 -lt 107374182458 2024-02-05T05:41:54.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:54.375 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:54.375 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-352187318274 2024-02-05T05:41:54.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:41:54.377 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-352187318274 2024-02-05T05:41:54.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:54.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-02-05T05:41:54.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 352187318274' 2024-02-05T05:41:54.379 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.5 seq 352187318274 2024-02-05T05:41:54.380 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:41:54.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318275 -lt 352187318274 2024-02-05T05:41:54.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:54.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855403 2024-02-05T05:41:54.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:54.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:41:54.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:54.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855403 2024-02-05T05:41:54.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855403 2024-02-05T05:41:54.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855403' 2024-02-05T05:41:54.740 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.6 seq 150323855403 2024-02-05T05:41:54.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:41:55.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855404 -lt 150323855403 2024-02-05T05:41:55.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:55.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691878 2024-02-05T05:41:55.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:55.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:41:55.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691878 2024-02-05T05:41:55.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:55.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691878 2024-02-05T05:41:55.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691878' 2024-02-05T05:41:55.103 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.7 seq 171798691878 2024-02-05T05:41:55.103 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:41:55.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691878 -lt 171798691878 2024-02-05T05:41:55.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:55.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528348 2024-02-05T05:41:55.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:55.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:41:55.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528348 2024-02-05T05:41:55.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:55.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528348 2024-02-05T05:41:55.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528348' 2024-02-05T05:41:55.439 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.8 seq 193273528348 2024-02-05T05:41:55.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:41:55.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528349 -lt 193273528348 2024-02-05T05:41:55.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:55.758 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481798 2024-02-05T05:41:55.758 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:55.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:41:55.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481798 2024-02-05T05:41:55.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:55.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-02-05T05:41:55.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-02-05T05:41:55.763 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.9 seq 330712481798 2024-02-05T05:41:55.763 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:41:56.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481799 -lt 330712481798 2024-02-05T05:41:56.102 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:41:56.102 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:56.103 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:56.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:41:56.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:41:56.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:41:56.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:41:56.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:41:56.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:41:56.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:41:56.488 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:41:56.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:41:56.801 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:41:56.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:56.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:57.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:41:57.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:41:57.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:41:57.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:41:57.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:41:57.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:41:57.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:41:57.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:41:57.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:41:57.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:41:57.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:41:57.488 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:41:57.488 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:41:57.488 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:41:57.488 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:41:57.488 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:41:57.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:41:22.304157+0000 2024-02-05T05:41:57.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:41:58.100 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:41:58.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:41:22.304157+0000 2024-02-05T05:41:58.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:41:58.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:41:22.304157+0000 2024-02-05T05:41:58.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:41:58.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:41:58.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:41:58.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:41:58.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:41:58.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:41:58.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:41:58.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:41:58.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:41:22.304157+0000 '>' 2024-02-05T05:41:22.304157+0000 2024-02-05T05:41:58.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:41:59.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:41:59.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:41:59.424 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:41:59.424 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:41:59.424 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:41:59.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:41:59.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:41:59.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:41:22.304157+0000 '>' 2024-02-05T05:41:22.304157+0000 2024-02-05T05:41:59.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:42:00.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:42:00.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:42:00.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:42:00.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:42:00.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:42:00.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:42:00.738 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:42:01.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:41:58.435767+0000 '>' 2024-02-05T05:41:22.304157+0000 2024-02-05T05:42:01.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:42:01.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:42:01.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:42:01.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:42:01.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:42:01.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:42:01.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:42:01.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:42:01.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:42:01.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:42:01.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:42:01.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:42:01.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:42:01.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:42:01.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:42:01.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:42:01.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:42:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:42:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:42:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:42:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:42:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:42:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:42:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:42:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-02-05T05:42:03.585 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:42:03.586 INFO:tasks.workunit.client.0.smithi148.stdout:hinfo_key 2024-02-05T05:42:03.586 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:42:04.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:42:04.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:42:04.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:42:04.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:42:04.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:42:04.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:42:04.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:42:04.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:42:04.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:42:04.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:42:04.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:42:04.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:42:04.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:42:04.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:42:04.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:42:04.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:42:04.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:42:04.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:42:04.142 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:42:04.142 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:42:04.142 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:42:04.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:42:04.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:42:04.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:42:04.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:42:04.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:42:04.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:42:04.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:42:04.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:42:04.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:42:04.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:42:04.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:42:04.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:42:04.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:42:04.144 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.9 2024-02-05T05:42:04.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:42:04.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:42:04.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:42:04.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:42:04.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:42:04.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:42:04.177 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:42:04.176+0000 7ff2cc6c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:04.180 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:42:04.180+0000 7ff2cc6c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:04.182 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:42:04.180+0000 7ff2cc6c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:04.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:42:04.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:42:04.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:42:04.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:42:04.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:42:04.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:04.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:42:04.470 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:42:04.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:04.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:42:04.718 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:42:04.716+0000 7ff2cc6c2640 -1 Falling back to public interface 2024-02-05T05:42:04.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:05.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:05.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:05.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:42:05.817 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:42:05.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:05.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:42:06.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:07.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:07.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:07.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:42:07.157 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:42:07.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:07.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:42:07.212 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:42:07.212+0000 7ff2cc6c2640 -1 osd.9 84 log_to_monitors true 2024-02-05T05:42:07.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:08.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:08.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:08.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:42:08.529 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:42:08.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:08.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:42:09.015 INFO:tasks.workunit.client.0.smithi148.stderr:osd.9 up in weight 1 up_from 88 up_thru 0 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6874/391132626,v1:127.0.0.1:6875/391132626] [v2:127.0.0.1:6876/391132626,v1:127.0.0.1:6877/391132626] exists,up c49238e7-ce58-439e-bf47-f8337ce62570 2024-02-05T05:42:09.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:42:09.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:42:09.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:42:09.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:42:09.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:42:09.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:42:09.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:42:09.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:42:09.016 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:42:09.016 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:42:09.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:42:09.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:42:09.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:42:09.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:42:09.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:42:09.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:42:09.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:42:09.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:42:09.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:42:09.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:42:09.698 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:42:09.698 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:42:09.698 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:42:09.698 INFO:tasks.workunit.client.0.smithi148.stderr:4 2024-02-05T05:42:09.698 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:42:09.699 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:42:09.699 INFO:tasks.workunit.client.0.smithi148.stderr:7 2024-02-05T05:42:09.699 INFO:tasks.workunit.client.0.smithi148.stderr:8 2024-02-05T05:42:09.699 INFO:tasks.workunit.client.0.smithi148.stderr:9' 2024-02-05T05:42:09.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:42:09.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:09.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:42:10.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836574 2024-02-05T05:42:10.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836574 2024-02-05T05:42:10.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574' 2024-02-05T05:42:10.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:10.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:42:10.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673046 2024-02-05T05:42:10.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673046 2024-02-05T05:42:10.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673046' 2024-02-05T05:42:10.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:10.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:42:10.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509519 2024-02-05T05:42:10.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509519 2024-02-05T05:42:10.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673046 2-64424509519' 2024-02-05T05:42:10.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:10.705 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:42:10.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678032 2024-02-05T05:42:10.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678032 2024-02-05T05:42:10.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673046 2-64424509519 3-304942678032' 2024-02-05T05:42:10.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:10.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:42:11.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182463 2024-02-05T05:42:11.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182463 2024-02-05T05:42:11.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673046 2-64424509519 3-304942678032 4-107374182463' 2024-02-05T05:42:11.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:11.038 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:42:11.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318279 2024-02-05T05:42:11.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318279 2024-02-05T05:42:11.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673046 2-64424509519 3-304942678032 4-107374182463 5-352187318279' 2024-02-05T05:42:11.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:11.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:42:11.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855408 2024-02-05T05:42:11.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855408 2024-02-05T05:42:11.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673046 2-64424509519 3-304942678032 4-107374182463 5-352187318279 6-150323855408' 2024-02-05T05:42:11.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:11.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:42:11.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691883 2024-02-05T05:42:11.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691883 2024-02-05T05:42:11.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673046 2-64424509519 3-304942678032 4-107374182463 5-352187318279 6-150323855408 7-171798691883' 2024-02-05T05:42:11.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:11.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:42:11.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528353 2024-02-05T05:42:11.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528353 2024-02-05T05:42:11.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673046 2-64424509519 3-304942678032 4-107374182463 5-352187318279 6-150323855408 7-171798691883 8-193273528353' 2024-02-05T05:42:11.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:11.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:42:11.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-02-05T05:42:11.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-02-05T05:42:11.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673046 2-64424509519 3-304942678032 4-107374182463 5-352187318279 6-150323855408 7-171798691883 8-193273528353 9-377957122050' 2024-02-05T05:42:11.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:11.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836574 2024-02-05T05:42:11.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:11.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:42:11.756 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836574 2024-02-05T05:42:11.756 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:11.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836574 2024-02-05T05:42:11.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-02-05T05:42:11.757 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836574 2024-02-05T05:42:11.757 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:12.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836573 -lt 21474836574 2024-02-05T05:42:12.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:42:13.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:42:13.075 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:13.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836575 -lt 21474836574 2024-02-05T05:42:13.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:13.402 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673046 2024-02-05T05:42:13.402 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:13.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:42:13.404 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673046 2024-02-05T05:42:13.405 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:13.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673046 2024-02-05T05:42:13.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-02-05T05:42:13.407 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949673046 2024-02-05T05:42:13.407 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:42:13.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673046 -lt 42949673046 2024-02-05T05:42:13.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:13.739 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509519 2024-02-05T05:42:13.739 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:13.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:42:13.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509519 2024-02-05T05:42:13.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:13.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509519 2024-02-05T05:42:13.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509519' 2024-02-05T05:42:13.743 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509519 2024-02-05T05:42:13.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:42:14.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509520 -lt 64424509519 2024-02-05T05:42:14.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:14.091 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678032 2024-02-05T05:42:14.092 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:14.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:42:14.094 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678032 2024-02-05T05:42:14.094 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:14.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678032 2024-02-05T05:42:14.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678032' 2024-02-05T05:42:14.096 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 304942678032 2024-02-05T05:42:14.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:42:14.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678033 -lt 304942678032 2024-02-05T05:42:14.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:14.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182463 2024-02-05T05:42:14.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:14.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:42:14.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182463 2024-02-05T05:42:14.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:14.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182463 2024-02-05T05:42:14.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-02-05T05:42:14.445 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.4 seq 107374182463 2024-02-05T05:42:14.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:42:14.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182464 -lt 107374182463 2024-02-05T05:42:14.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:14.809 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-352187318279 2024-02-05T05:42:14.809 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:14.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:42:14.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-352187318279 2024-02-05T05:42:14.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318279 2024-02-05T05:42:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 352187318279' 2024-02-05T05:42:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.5 seq 352187318279 2024-02-05T05:42:14.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:42:15.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318280 -lt 352187318279 2024-02-05T05:42:15.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:15.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855408 2024-02-05T05:42:15.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:15.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:42:15.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855408 2024-02-05T05:42:15.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:15.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855408 2024-02-05T05:42:15.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855408' 2024-02-05T05:42:15.186 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.6 seq 150323855408 2024-02-05T05:42:15.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:42:15.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855409 -lt 150323855408 2024-02-05T05:42:15.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:15.501 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691883 2024-02-05T05:42:15.501 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:15.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:42:15.504 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691883 2024-02-05T05:42:15.504 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:15.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691883 2024-02-05T05:42:15.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-02-05T05:42:15.506 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.7 seq 171798691883 2024-02-05T05:42:15.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:42:15.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691883 -lt 171798691883 2024-02-05T05:42:15.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:15.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528353 2024-02-05T05:42:15.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:15.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:42:15.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528353 2024-02-05T05:42:15.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:15.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528353 2024-02-05T05:42:15.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528353' 2024-02-05T05:42:15.814 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.8 seq 193273528353 2024-02-05T05:42:15.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:42:16.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528354 -lt 193273528353 2024-02-05T05:42:16.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:16.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-377957122050 2024-02-05T05:42:16.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:16.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:42:16.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-377957122050 2024-02-05T05:42:16.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:16.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-02-05T05:42:16.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2024-02-05T05:42:16.207 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.9 seq 377957122050 2024-02-05T05:42:16.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:42:16.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122051 -lt 377957122050 2024-02-05T05:42:16.545 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:42:16.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:16.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:16.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:42:16.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:42:16.970 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:42:16.970 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:42:16.970 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:42:16.970 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:42:16.970 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:42:16.970 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:42:17.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:42:17.332 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:42:17.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:17.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:17.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:42:17.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:42:17.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:42:17.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:42:17.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:42:17.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-02-05T05:42:17.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:42:17.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:42:17.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-02-05T05:42:17.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-02-05T05:42:17.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:42:17.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:42:17.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:42:17.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:42:17.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 221139"' 2024-02-05T05:42:17.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 221139' 2024-02-05T05:42:17.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:42:17.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:42:17.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:42:17.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:42:17.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/221141: /' 2024-02-05T05:42:17.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 221142"' 2024-02-05T05:42:17.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 221142' 2024-02-05T05:42:17.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:42:17.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 221139 221142' 2024-02-05T05:42:17.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:42:17.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:42:17.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 221139 2024-02-05T05:42:17.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:42:17.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/221145: /' 2024-02-05T05:42:22.166 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:42:22.166 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:42:22.166 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:42:22.166 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:42:22.166 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:42:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:42:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:42:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:42:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:42:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:42:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:42:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:42:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:42:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:42:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:42:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:42:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:42:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:42:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:42:22.168 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:42:22.168 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:42:22.168 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:42:22.168 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-02-05T05:42:22.168 INFO:tasks.workunit.client.0.smithi148.stderr:221145: remove 2#2:eb822e21:::SOMETHING:head# 2024-02-05T05:42:22.168 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:42:22.168 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:42:22.168 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:42:22.168 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:42:22.168 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:42:22.168 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:42:22.168 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:42:22.168 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:42:22.171 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:42:22.171 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:42:22.171 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:42:22.171 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:42:22.171 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:42:22.172 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:42:22.172 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:42:22.172 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:42:22.172 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:42:22.172 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:42:22.172 INFO:tasks.workunit.client.0.smithi148.stderr:221145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:42:22.172 INFO:tasks.workunit.client.0.smithi148.stderr:221145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:42:22.172 INFO:tasks.workunit.client.0.smithi148.stderr:221145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:42:22.172 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:42:22.172 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:42:22.172 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:42:22.172 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:42:22.172 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:42:22.173 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:42:22.173 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:42:22.173 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:42:22.173 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:42:22.173 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:42:22.173 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:42:22.173 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:42:22.173 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:42:22.173 INFO:tasks.workunit.client.0.smithi148.stderr:221145: start osd.9 2024-02-05T05:42:22.173 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:42:22.322 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:42:22.322 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:42:22.323 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:42:22.323 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:42:22.323 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:42:22.323 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:42:22.323 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:42:22.323 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:42:22.323 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:42:22.323 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:42:22.323 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:42:22.323 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:42:22.323 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:42:22.323 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:42:22.323 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:42:22.324 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:42:22.324 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:42:22.324 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:42:22.324 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:42:22.324 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:42:22.324 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:42:22.324 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:42:22.324 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-02-05T05:42:22.324 INFO:tasks.workunit.client.0.smithi148.stderr:221141: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:42:22.324 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:42:22.324 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:42:22.324 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:42:22.324 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:42:22.325 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:42:22.325 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:42:22.325 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:42:22.325 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:42:22.325 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:42:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:42:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:42:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:42:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:42:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:42:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:42:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:42:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:42:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:42:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:221141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:42:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:221141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:42:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:221141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:42:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:42:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:42:22.327 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:42:22.327 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:42:22.327 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:42:22.327 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:42:22.327 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:42:22.327 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:42:22.327 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:42:22.327 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:42:22.327 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:42:22.327 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:42:22.327 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:42:22.327 INFO:tasks.workunit.client.0.smithi148.stderr:221141: start osd.5 2024-02-05T05:42:22.327 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:42:26.909 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:42:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:42:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:42:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:42:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:42:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 2024-02-05T05:42:22.203+0000 7f8131b60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 2024-02-05T05:42:22.207+0000 7f8131b60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 2024-02-05T05:42:22.207+0000 7f8131b60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:42:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:42:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:42:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:42:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:42:26.911 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:26.911 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:42:26.911 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 0 2024-02-05T05:42:26.911 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:26.911 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:42:26.911 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 2024-02-05T05:42:22.755+0000 7f8131b60640 -1 Falling back to public interface 2024-02-05T05:42:26.911 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:26.911 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:26.911 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:26.911 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:42:26.911 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 1 2024-02-05T05:42:26.911 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:26.911 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:42:26.911 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:26.912 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:26.912 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:26.912 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:42:26.912 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 2 2024-02-05T05:42:26.912 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:26.912 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:42:26.912 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:26.912 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 2024-02-05T05:42:25.607+0000 7f8131b60640 -1 osd.9 90 log_to_monitors true 2024-02-05T05:42:26.912 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:26.912 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:26.912 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:42:26.912 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 3 2024-02-05T05:42:26.912 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:26.913 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:42:27.094 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:42:27.094 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:42:27.094 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:42:27.094 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:42:27.094 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:42:27.095 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 2024-02-05T05:42:22.351+0000 7f135221e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:27.095 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 2024-02-05T05:42:22.363+0000 7f135221e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:27.095 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 2024-02-05T05:42:22.367+0000 7f135221e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:27.095 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:42:27.095 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:42:27.095 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:42:27.095 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:42:27.095 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:42:27.095 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:27.095 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:42:27.095 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 0 2024-02-05T05:42:27.095 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:27.095 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:42:27.096 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 2024-02-05T05:42:22.915+0000 7f135221e640 -1 Falling back to public interface 2024-02-05T05:42:27.096 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:27.096 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:27.096 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:27.096 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:42:27.096 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 1 2024-02-05T05:42:27.096 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:27.096 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:42:27.096 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:27.096 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:27.096 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:27.096 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:42:27.096 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 2 2024-02-05T05:42:27.097 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:27.097 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:42:27.097 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:27.097 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 2024-02-05T05:42:26.307+0000 7f135221e640 -1 osd.5 90 log_to_monitors true 2024-02-05T05:42:27.097 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:27.097 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:27.097 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:42:27.097 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 3 2024-02-05T05:42:27.097 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:27.097 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:42:29.679 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:29.679 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:29.679 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:29.679 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:42:29.679 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 4 2024-02-05T05:42:29.680 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:29.680 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:42:29.680 INFO:tasks.workunit.client.0.smithi148.stderr:221145: osd.9 up in weight 1 up_from 95 up_thru 0 down_at 91 last_clean_interval [88,90) [v2:127.0.0.1:6842/1373687744,v1:127.0.0.1:6843/1373687744] [v2:127.0.0.1:6844/1373687744,v1:127.0.0.1:6845/1373687744] exists,up c49238e7-ce58-439e-bf47-f8337ce62570 2024-02-05T05:42:29.680 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:42:29.680 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:42:29.680 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:42:29.680 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:42:29.680 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:42:29.680 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:42:29.680 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:42:29.680 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:42:29.680 INFO:tasks.workunit.client.0.smithi148.stderr:221145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:42:29.681 INFO:tasks.workunit.client.0.smithi148.stderr:221145: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:42:29.681 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:42:29.681 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:42:29.681 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:42:29.681 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:42:29.681 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:42:29.681 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:42:29.681 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:42:29.681 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:42:29.681 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:42:29.681 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:42:29.681 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 1 2024-02-05T05:42:29.681 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 2 2024-02-05T05:42:29.681 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 3 2024-02-05T05:42:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 4 2024-02-05T05:42:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 5 2024-02-05T05:42:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 6 2024-02-05T05:42:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 7 2024-02-05T05:42:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 8 2024-02-05T05:42:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:221145: 9' 2024-02-05T05:42:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:42:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:42:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836579 2024-02-05T05:42:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836579 2024-02-05T05:42:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579' 2024-02-05T05:42:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /s.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:29.683 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:29.683 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:29.683 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:42:29.683 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 4 2024-02-05T05:42:29.683 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:29.683 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:42:29.683 INFO:tasks.workunit.client.0.smithi148.stderr:221141: osd.5 up in weight 1 up_from 96 up_thru 82 down_at 91 last_clean_interval [82,90) [v2:127.0.0.1:6874/1926191796,v1:127.0.0.1:6875/1926191796] [v2:127.0.0.1:6876/1926191796,v1:127.0.0.1:6877/1926191796] exists,up f12201fc-c63f-4591-a81c-5c4646aaab07 2024-02-05T05:42:29.683 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:42:29.683 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:42:29.683 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:42:29.683 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:42:29.683 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:42:29.683 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:42:29.683 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:42:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:42:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:221141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:42:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:221141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:42:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:42:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:42:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:42:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:42:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:42:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:42:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:42:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:42:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:42:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:42:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 1 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 2 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 3 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 4 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 5 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 6 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 7 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 8 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: 9' 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836580 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836580 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580' 2024-02-05T05:42:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.337 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:42:31.337 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673051 2024-02-05T05:42:31.338 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673051 2024-02-05T05:42:31.338 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673051' 2024-02-05T05:42:31.338 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.338 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:42:31.338 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509525 2024-02-05T05:42:31.338 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509525 2024-02-05T05:42:31.338 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673051 2-64424509525' 2024-02-05T05:42:31.338 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.338 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:42:31.338 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678038 2024-02-05T05:42:31.338 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678038 2024-02-05T05:42:31.338 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673051 2-64424509525 3-304942678038' 2024-02-05T05:42:31.338 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.339 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:42:31.339 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182469 2024-02-05T05:42:31.339 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182469 2024-02-05T05:42:31.339 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673051 2-64424509525 3-304942678038 4-107374182469' 2024-02-05T05:42:31.339 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.339 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:42:31.339 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-02-05T05:42:31.339 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-02-05T05:42:31.339 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673051 2-64424509525 3-304942678038 4-107374182469 5-412316860418' 2024-02-05T05:42:31.339 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.339 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:42:31.339 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855413 2024-02-05T05:42:31.339 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855413 2024-02-05T05:42:31.339 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673051 2-64424509525 3-304942678038 4-107374182469 5-412316860418 6-150323855413' 2024-02-05T05:42:31.340 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.340 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:42:31.341 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:42:31.341 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673052 2024-02-05T05:42:31.341 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673052 2024-02-05T05:42:31.341 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673052' 2024-02-05T05:42:31.341 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.341 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:42:31.341 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509524 2024-02-05T05:42:31.341 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509524 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673052 2-64424509524' 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678037 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678037 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673052 2-64424509524 3-304942678037' 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182468 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182468 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673052 2-64424509524 3-304942678037 4-107374182468' 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860419 2024-02-05T05:42:31.342 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860419 2024-02-05T05:42:31.343 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673052 2-64424509524 3-304942678037 4-107374182468 5-412316860419' 2024-02-05T05:42:31.343 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.343 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:42:31.343 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855414 2024-02-05T05:42:31.343 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855414 2024-02-05T05:42:31.343 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673052 2-64424509524 3-304942678037 4-107374182468 5-412316860419 6-150323855414' 2024-02-05T05:42:31.343 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.343 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:42:31.902 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flushpg_stats: seq=171798691888 2024-02-05T05:42:31.902 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691888 2024-02-05T05:42:31.902 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673051 2-64424509525 3-304942678038 4-107374182469 5-412316860418 6-150323855413 7-171798691888' 2024-02-05T05:42:31.902 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.902 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:42:31.902 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528358 2024-02-05T05:42:31.902 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528358 2024-02-05T05:42:31.902 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673051 2-64424509525 3-304942678038 4-107374182469 5-412316860418 6-150323855413 7-171798691888 8-193273528358' 2024-02-05T05:42:31.902 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.902 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:42:31.902 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893123 2024-02-05T05:42:31.903 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893123 2024-02-05T05:42:31.903 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673051 2-64424509525 3-304942678038 4-107374182469 5-412316860418 6-150323855413 7-171798691888 8-193273528358 9-408021893123' 2024-02-05T05:42:31.903 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:31.903 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836579 2024-02-05T05:42:31.903 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:31.903 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:42:31.903 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836579 2024-02-05T05:42:31.903 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:31.903 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836579 2024-02-05T05:42:31.903 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-02-05T05:42:31.903 INFO:tasks.workunit.client.0.smithi148.stderr:221145: waiting osd.0 seq 21474836579 2024-02-05T05:42:31.903 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:31.903 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836580 -lt 21474836579 2024-02-05T05:42:31.904 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:31.904 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673051 2024-02-05T05:42:31.904 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:31.904 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:42:31.904 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673051 2024-02-05T05:42:31.904 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:31.904 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673051 2024-02-05T05:42:31.904 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-02-05T05:42:31.904 INFO:tasks.workunit.client.0.smithi148.stderr:221145: waiting osd.1 seq 42949673051 2024-02-05T05:42:31.907 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_s_pg_stats: seq=171798691889 2024-02-05T05:42:31.907 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691889 2024-02-05T05:42:31.907 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673052 2-64424509524 3-304942678037 4-107374182468 5-412316860419 6-150323855414 7-171798691889' 2024-02-05T05:42:31.907 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.907 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:42:31.907 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528359 2024-02-05T05:42:31.907 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528359 2024-02-05T05:42:31.907 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673052 2-64424509524 3-304942678037 4-107374182468 5-412316860419 6-150323855414 7-171798691889 8-193273528359' 2024-02-05T05:42:31.907 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:31.907 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:42:31.907 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893124 2024-02-05T05:42:31.907 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893124 2024-02-05T05:42:31.907 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673052 2-64424509524 3-304942678037 4-107374182468 5-412316860419 6-150323855414 7-171798691889 8-193273528359 9-408021893124' 2024-02-05T05:42:31.907 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:31.908 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836580 2024-02-05T05:42:31.908 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:31.908 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:42:31.908 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836580 2024-02-05T05:42:31.908 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:31.908 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836580 2024-02-05T05:42:31.908 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-02-05T05:42:31.908 INFO:tasks.workunit.client.0.smithi148.stderr:221141: waiting osd.0 seq 21474836580 2024-02-05T05:42:31.908 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:31.908 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836580 -lt 21474836580 2024-02-05T05:42:31.908 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:31.908 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673052 2024-02-05T05:42:31.908 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:31.909 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:42:31.909 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673052 2024-02-05T05:42:31.909 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:31.909 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673052 2024-02-05T05:42:31.909 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-02-05T05:42:31.909 INFO:tasks.workunit.client.0.smithi148.stderr:221141: waiting osd.1 seq 42949673052 2024-02-05T05:42:33.125 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_tats: ceph osd last-stat-seq 1 2024-02-05T05:42:33.125 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673051 2024-02-05T05:42:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509525 2024-02-05T05:42:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:42:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509525 2024-02-05T05:42:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509525 2024-02-05T05:42:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-02-05T05:42:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:221145: waiting osd.2 seq 64424509525 2024-02-05T05:42:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:42:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509526 -lt 64424509525 2024-02-05T05:42:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678038 2024-02-05T05:42:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:42:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678038 2024-02-05T05:42:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678038 2024-02-05T05:42:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678038' 2024-02-05T05:42:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:221145: waiting osd.3 seq 304942678038 2024-02-05T05:42:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:42:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678038 -lt 304942678038 2024-02-05T05:42:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182469 2024-02-05T05:42:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:42:33.128 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182469 2024-02-05T05:42:33.128 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:33.128 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182469 2024-02-05T05:42:33.128 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182469' 2024-02-05T05:42:33.128 INFO:tasks.workunit.client.0.smithi148.stderr:221145: waiting osd.4 seq 107374182469 2024-02-05T05:42:33.128 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:42:33.128 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182469 -lt 107374182469 2024-02-05T05:42:33.128 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:33.128 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-412316860418 2024-02-05T05:42:33.217 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cestats: ceph osd last-stat-seq 1 2024-02-05T05:42:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673052 2024-02-05T05:42:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509524 2024-02-05T05:42:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:42:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509524 2024-02-05T05:42:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509524 2024-02-05T05:42:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-02-05T05:42:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:221141: waiting osd.2 seq 64424509524 2024-02-05T05:42:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:42:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509526 -lt 64424509524 2024-02-05T05:42:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678037 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678037 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678037 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678037' 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: waiting osd.3 seq 304942678037 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678038 -lt 304942678037 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182468 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:42:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182468 2024-02-05T05:42:33.220 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:33.220 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182468 2024-02-05T05:42:33.220 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182468' 2024-02-05T05:42:33.220 INFO:tasks.workunit.client.0.smithi148.stderr:221141: waiting osd.4 seq 107374182468 2024-02-05T05:42:33.220 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:42:33.220 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182469 -lt 107374182468 2024-02-05T05:42:33.220 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:33.220 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:34.093 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtesphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:34.093 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:42:34.094 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-412316860418 2024-02-05T05:42:34.094 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:34.094 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-02-05T05:42:34.094 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 412316860418' 2024-02-05T05:42:34.094 INFO:tasks.workunit.client.0.smithi148.stderr:221145: waiting osd.5 seq 412316860418 2024-02-05T05:42:34.094 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:42:34.094 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860418 2024-02-05T05:42:34.094 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:34.094 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855413 2024-02-05T05:42:34.094 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:34.094 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:42:34.094 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855413 2024-02-05T05:42:34.094 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855413 2024-02-05T05:42:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855413' 2024-02-05T05:42:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:221145: waiting osd.6 seq 150323855413 2024-02-05T05:42:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:42:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855414 -lt 150323855413 2024-02-05T05:42:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691888 2024-02-05T05:42:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:42:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691888 2024-02-05T05:42:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691888 2024-02-05T05:42:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691888' 2024-02-05T05:42:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:221145: waiting osd.7 seq 171798691888 2024-02-05T05:42:34.096 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:42:34.096 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691889 -lt 171798691888 2024-02-05T05:42:34.096 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:34.096 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528358 2024-02-05T05:42:34.096 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:34.096 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:42:34.096 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528358 2024-02-05T05:42:34.229 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: ct/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-412316860419 2024-02-05T05:42:34.229 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:42:34.229 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-412316860419 2024-02-05T05:42:34.229 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:34.229 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860419 2024-02-05T05:42:34.229 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 412316860419' 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: waiting osd.5 seq 412316860419 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860419 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855414 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855414 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855414 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855414' 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: waiting osd.6 seq 150323855414 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855414 -lt 150323855414 2024-02-05T05:42:34.230 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:34.231 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691889 2024-02-05T05:42:34.231 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:34.231 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:42:34.231 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691889 2024-02-05T05:42:34.231 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:34.231 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691889 2024-02-05T05:42:34.231 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691889' 2024-02-05T05:42:34.231 INFO:tasks.workunit.client.0.smithi148.stderr:221141: waiting osd.7 seq 171798691889 2024-02-05T05:42:34.231 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:42:34.231 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691889 -lt 171798691889 2024-02-05T05:42:34.231 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:34.231 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528359 2024-02-05T05:42:34.231 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:34.232 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:42:34.232 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528359 2024-02-05T05:42:35.858 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: ut -d - -f 2 2024-02-05T05:42:35.858 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528358 2024-02-05T05:42:35.858 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2024-02-05T05:42:35.858 INFO:tasks.workunit.client.0.smithi148.stderr:221145: waiting osd.8 seq 193273528358 2024-02-05T05:42:35.858 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:42:35.858 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528360 -lt 193273528358 2024-02-05T05:42:35.858 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:35.858 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-408021893123 2024-02-05T05:42:35.858 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:35.858 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:42:35.858 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-408021893123 2024-02-05T05:42:35.859 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:35.859 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893123 2024-02-05T05:42:35.859 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 408021893123' 2024-02-05T05:42:35.859 INFO:tasks.workunit.client.0.smithi148.stderr:221145: waiting osd.9 seq 408021893123 2024-02-05T05:42:35.859 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:42:35.859 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893123 2024-02-05T05:42:35.859 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:42:35.859 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:35.859 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:35.859 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:42:35.859 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:42:35.859 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:42:35.859 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:42:35.859 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:42:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:42:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:42:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:42:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:42:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:42:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:221145: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:42:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:42:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:221145: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:42:35.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:42:36.049 INFO:tasks.workunit.client.0.smithi148.stderr:cut -d - -f 2 2024-02-05T05:42:36.049 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528359 2024-02-05T05:42:36.049 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528359' 2024-02-05T05:42:36.049 INFO:tasks.workunit.client.0.smithi148.stderr:221141: waiting osd.8 seq 193273528359 2024-02-05T05:42:36.049 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:42:36.049 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528360 -lt 193273528359 2024-02-05T05:42:36.049 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:36.049 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-408021893124 2024-02-05T05:42:36.049 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:36.049 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:42:36.049 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-408021893124 2024-02-05T05:42:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893124 2024-02-05T05:42:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 408021893124' 2024-02-05T05:42:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:221141: waiting osd.9 seq 408021893124 2024-02-05T05:42:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:42:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893124 2024-02-05T05:42:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:42:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:42:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:42:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:42:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:42:36.050 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:42:36.051 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:42:36.051 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:42:36.051 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:42:36.051 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:42:36.051 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:42:36.051 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:36.051 INFO:tasks.workunit.client.0.smithi148.stderr:221141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:36.051 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:42:36.051 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:42:36.051 INFO:tasks.workunit.client.0.smithi148.stderr:221141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:42:36.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:42:36.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:42:36.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 221142 2024-02-05T05:42:36.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:42:36.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:42:36.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:42:36.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:42:36.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:42:36.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:42:36.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:42:36.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:42:36.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:42:36.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:42:36.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:42:36.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:42:36.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:42:36.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:42:36.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:42:36.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:42:36.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:42:36.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:42:36.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:41:58.435767+0000 2024-02-05T05:42:36.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:42:36.939 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:42:36.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:41:58.435767+0000 2024-02-05T05:42:36.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:42:36.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:41:58.435767+0000 2024-02-05T05:42:36.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:42:36.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:42:36.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:42:36.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:42:36.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:42:36.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:42:36.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:42:36.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:42:37.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:41:58.435767+0000 '>' 2024-02-05T05:41:58.435767+0000 2024-02-05T05:42:37.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:42:38.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:42:38.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:42:38.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:42:38.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:42:38.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:42:38.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:42:38.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:42:38.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:41:58.435767+0000 '>' 2024-02-05T05:41:58.435767+0000 2024-02-05T05:42:38.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:42:39.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:42:39.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:42:39.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:42:39.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:42:39.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:42:39.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:42:39.586 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:42:39.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:41:58.435767+0000 '>' 2024-02-05T05:41:58.435767+0000 2024-02-05T05:42:39.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:42:40.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:42:40.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:42:40.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:42:40.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:42:40.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:42:40.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:42:40.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:42:41.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:42:37.329889+0000 '>' 2024-02-05T05:41:58.435767+0000 2024-02-05T05:42:41.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:42:41.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:42:41.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:42:41.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:42:41.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:42:41.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 224570"' 2024-02-05T05:42:41.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 224570' 2024-02-05T05:42:41.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:42:41.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:42:41.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:42:41.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 224571"' 2024-02-05T05:42:41.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 224571' 2024-02-05T05:42:41.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:42:41.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 224570 224571' 2024-02-05T05:42:41.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:42:41.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:42:41.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 224570 2024-02-05T05:42:41.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/224574: /' 2024-02-05T05:42:41.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:42:41.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:42:41.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/224575: /' 2024-02-05T05:42:44.896 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:42:44.896 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:42:44.896 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:42:44.896 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:42:44.896 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:42:44.896 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:42:44.896 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:42:44.896 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:42:44.896 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:42:44.897 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:42:44.897 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:42:44.897 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:42:44.897 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:42:44.897 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:42:44.897 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:42:44.897 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:42:44.897 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:42:44.897 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:42:44.897 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:42:44.897 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:42:44.897 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:42:44.897 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:42:44.897 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-02-05T05:42:44.898 INFO:tasks.workunit.client.0.smithi148.stderr:224575: _ 2024-02-05T05:42:44.898 INFO:tasks.workunit.client.0.smithi148.stderr:224575: hinfo_key 2024-02-05T05:42:44.898 INFO:tasks.workunit.client.0.smithi148.stderr:224575: snapset 2024-02-05T05:42:44.898 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:42:44.898 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:42:44.898 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:42:44.898 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:42:44.898 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:42:44.898 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:42:44.898 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:42:44.898 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:42:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:42:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:42:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:42:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:42:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:42:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:42:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:42:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:42:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:42:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:42:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:224575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:42:44.902 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:42:44.903 INFO:tasks.workunit.client.0.smithi148.stderr:224575: start osd.9 2024-02-05T05:42:45.226 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:42:45.226 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:42:45.226 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:42:45.226 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:42:45.226 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:42:45.227 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:42:45.227 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:42:45.227 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:42:45.227 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:42:45.227 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:42:45.227 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:42:45.227 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:42:45.227 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:42:45.227 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:42:45.227 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:42:45.227 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:42:45.227 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:42:45.227 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:42:45.227 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:42:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:42:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:42:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:42:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-02-05T05:42:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:224574: _ 2024-02-05T05:42:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:224574: hinfo_key 2024-02-05T05:42:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:224574: snapset 2024-02-05T05:42:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:42:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:42:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:42:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:42:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:42:45.229 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:42:45.229 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:42:45.229 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:42:45.229 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:42:45.229 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:42:45.229 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:42:45.229 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:42:45.229 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:42:45.229 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:42:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:42:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:42:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:42:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:42:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:224574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:42:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:224574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:42:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:224574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:42:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:42:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:42:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:42:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:42:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:42:45.231 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:42:45.231 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:42:45.231 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:42:45.231 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:42:45.231 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:42:45.231 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:42:45.231 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:42:45.231 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:42:45.231 INFO:tasks.workunit.client.0.smithi148.stderr:224574: start osd.5 2024-02-05T05:42:49.711 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:42:49.711 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:42:49.711 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:42:49.711 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:42:49.711 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:42:49.712 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:42:49.712 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 2024-02-05T05:42:44.935+0000 7fd86d11a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:49.712 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 2024-02-05T05:42:44.935+0000 7fd86d11a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:49.712 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 2024-02-05T05:42:44.939+0000 7fd86d11a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:49.712 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:42:49.712 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:42:49.712 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:42:49.712 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:42:49.712 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:42:49.712 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:49.712 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:42:49.712 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 0 2024-02-05T05:42:49.712 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:49.713 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:42:49.713 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 2024-02-05T05:42:45.483+0000 7fd86d11a640 -1 Falling back to public interface 2024-02-05T05:42:49.713 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:49.713 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:49.713 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:49.713 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:42:49.713 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 1 2024-02-05T05:42:49.713 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:49.713 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:42:49.713 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:49.713 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:49.713 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:49.714 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:42:49.714 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 2 2024-02-05T05:42:49.714 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:49.714 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:42:49.714 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 2024-02-05T05:42:48.007+0000 7fd86d11a640 -1 osd.9 98 log_to_monitors true 2024-02-05T05:42:49.714 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:49.714 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:49.714 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:49.714 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:42:49.714 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 3 2024-02-05T05:42:49.714 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:49.714 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:42:50.157 INFO:tasks.workunit.client.0.smithi148.stderr:224575: osd.9 up in weight 1 up_from 103 up_thru 0 down_aovery_ops 2024-02-05T05:42:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:42:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:42:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:42:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:42:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:42:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 2024-02-05T05:42:45.255+0000 7f2cbc1f7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 2024-02-05T05:42:45.299+0000 7f2cbc1f7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 2024-02-05T05:42:45.299+0000 7f2cbc1f7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:42:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:42:50.158 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 0 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 2024-02-05T05:42:45.875+0000 7f2cbc1f7640 -1 Falling back to public interface 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 1 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:50.159 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 2 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 2024-02-05T05:42:48.999+0000 7f2cbc1f7640 -1 osd.5 98 log_to_monitors true 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 3 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:50.160 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:42:51.415 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ct 99 last_clean_interval [95,98) [v2:127.0.0.1:6842/2986026923,v1:127.0.0.1:6843/2986026923] [v2:127.0.0.1:6844/2986026923,v1:127.0.0.1:6845/2986026923] exists,up c49238e7-ce58-439e-bf47-f8337ce62570 2024-02-05T05:42:51.415 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:42:51.415 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:42:51.415 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:42:51.415 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:42:51.415 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:42:51.415 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:42:51.416 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:42:51.416 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:42:51.416 INFO:tasks.workunit.client.0.smithi148.stderr:224575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:42:51.416 INFO:tasks.workunit.client.0.smithi148.stderr:224575: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:42:51.416 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:42:51.416 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:42:51.416 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:42:51.416 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:42:51.416 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:42:51.416 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:42:51.416 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:42:51.416 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:42:51.417 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:42:51.417 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:42:51.417 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 1 2024-02-05T05:42:51.417 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 2 2024-02-05T05:42:51.417 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 3 2024-02-05T05:42:51.417 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 4 2024-02-05T05:42:51.417 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 5 2024-02-05T05:42:51.417 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 6 2024-02-05T05:42:51.417 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 7 2024-02-05T05:42:51.417 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 8 2024-02-05T05:42:51.417 INFO:tasks.workunit.client.0.smithi148.stderr:224575: 9' 2024-02-05T05:42:51.417 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:42:51.417 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:51.417 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:42:51.418 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836586 2024-02-05T05:42:51.418 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836586 2024-02-05T05:42:51.418 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586' 2024-02-05T05:42:51.418 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:51.418 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:42:51.418 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673057 2024-02-05T05:42:51.418 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673057 2024-02-05T05:42:51.418 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673057' 2024-02-05T05:42:51.418 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:52.799 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.eph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:52.800 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:52.800 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:52.800 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:42:52.800 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 4 2024-02-05T05:42:52.800 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:52.800 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:42:52.800 INFO:tasks.workunit.client.0.smithi148.stderr:224574: osd.5 up in weight 1 up_from 106 up_thru 106 down_at 99 last_clean_interval [96,98) [v2:127.0.0.1:6874/3671734443,v1:127.0.0.1:6875/3671734443] [v2:127.0.0.1:6876/3671734443,v1:127.0.0.1:6877/3671734443] exists,up f12201fc-c63f-4591-a81c-5c4646aaab07 2024-02-05T05:42:52.800 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:42:52.800 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:42:52.800 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:42:52.800 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:42:52.801 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:42:52.801 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:42:52.801 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:42:52.801 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:42:52.801 INFO:tasks.workunit.client.0.smithi148.stderr:224574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:42:52.801 INFO:tasks.workunit.client.0.smithi148.stderr:224574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:42:52.801 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:42:52.801 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:42:52.801 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:42:52.801 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:42:52.801 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:42:52.801 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:42:52.801 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:42:52.802 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:42:52.802 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:42:52.802 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:42:52.802 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 1 2024-02-05T05:42:52.802 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 2 2024-02-05T05:42:52.802 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 3 2024-02-05T05:42:52.802 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 4 2024-02-05T05:42:52.802 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 5 2024-02-05T05:42:52.802 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 6 2024-02-05T05:42:52.802 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 7 2024-02-05T05:42:52.802 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 8 2024-02-05T05:42:52.802 INFO:tasks.workunit.client.0.smithi148.stderr:224574: 9' 2024-02-05T05:42:52.802 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:42:52.802 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:52.803 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:42:52.803 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836587 2024-02-05T05:42:52.803 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836587 2024-02-05T05:42:52.803 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587' 2024-02-05T05:42:53.292 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:42:53.293 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509530 2024-02-05T05:42:53.293 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509530 2024-02-05T05:42:53.293 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673057 2-64424509530' 2024-02-05T05:42:53.293 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:53.293 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:42:53.293 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678043 2024-02-05T05:42:53.294 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678043 2024-02-05T05:42:53.294 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673057 2-64424509530 3-304942678043' 2024-02-05T05:42:53.294 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:53.294 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:42:53.294 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182474 2024-02-05T05:42:53.294 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182474 2024-02-05T05:42:53.294 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673057 2-64424509530 3-304942678043 4-107374182474' 2024-02-05T05:42:53.295 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:53.295 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:42:53.295 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533378 2024-02-05T05:42:53.295 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533378 2024-02-05T05:42:53.295 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673057 2-64424509530 3-304942678043 4-107374182474 5-455266533378' 2024-02-05T05:42:53.295 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:53.295 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:42:53.296 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855419 2024-02-05T05:42:53.296 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855419 2024-02-05T05:42:53.296 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673057 2-64424509530 3-304942678043 4-107374182474 5-455266533378 6-150323855419' 2024-02-05T05:42:53.296 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:53.296 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:42:53.296 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691894 2024-02-05T05:42:53.296 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691894 2024-02-05T05:42:53.297 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673057 2-64424509530 3-304942678043 4-107374182474 5-455266533378 6-150323855419 7-171798691894' 2024-02-05T05:42:53.297 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:53.297 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:42:53.866 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cepids 2024-02-05T05:42:53.866 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:42:53.866 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673058 2024-02-05T05:42:53.866 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673058 2024-02-05T05:42:53.866 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673058' 2024-02-05T05:42:53.866 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:53.866 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:42:53.867 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509532 2024-02-05T05:42:53.867 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509532 2024-02-05T05:42:53.867 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673058 2-64424509532' 2024-02-05T05:42:53.867 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:53.867 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:42:53.867 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678044 2024-02-05T05:42:53.867 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678044 2024-02-05T05:42:53.867 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673058 2-64424509532 3-304942678044' 2024-02-05T05:42:53.867 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:53.867 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:42:53.867 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182476 2024-02-05T05:42:53.868 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182476 2024-02-05T05:42:53.868 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673058 2-64424509532 3-304942678044 4-107374182476' 2024-02-05T05:42:53.868 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:53.868 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:42:53.868 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533379 2024-02-05T05:42:53.868 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533379 2024-02-05T05:42:53.868 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673058 2-64424509532 3-304942678044 4-107374182476 5-455266533379' 2024-02-05T05:42:53.868 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:53.868 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:42:53.868 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855421 2024-02-05T05:42:53.868 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855421 2024-02-05T05:42:53.869 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673058 2-64424509532 3-304942678044 4-107374182476 5-455266533379 6-150323855421' 2024-02-05T05:42:53.869 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:53.869 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:42:54.113 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528365 2024-02-05T05:42:54.113 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528365 2024-02-05T05:42:54.113 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673057 2-64424509530 3-304942678043 4-107374182474 5-455266533378 6-150323855419 7-171798691894 8-193273528365' 2024-02-05T05:42:54.113 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:54.113 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:42:54.114 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631491 2024-02-05T05:42:54.114 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631491 2024-02-05T05:42:54.114 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673057 2-64424509530 3-304942678043 4-107374182474 5-455266533378 6-150323855419 7-171798691894 8-193273528365 9-442381631491' 2024-02-05T05:42:54.114 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:54.114 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836586 2024-02-05T05:42:54.114 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:54.114 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:42:54.114 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836586 2024-02-05T05:42:54.114 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:54.114 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836586 2024-02-05T05:42:54.114 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-02-05T05:42:54.114 INFO:tasks.workunit.client.0.smithi148.stderr:224575: waiting osd.0 seq 21474836586 2024-02-05T05:42:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836586 -lt 21474836586 2024-02-05T05:42:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673057 2024-02-05T05:42:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:42:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673057 2024-02-05T05:42:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673057 2024-02-05T05:42:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-02-05T05:42:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:224575: waiting osd.1 seq 42949673057 2024-02-05T05:42:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:42:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673057 -lt 42949673057 2024-02-05T05:42:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:54.116 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509530 2024-02-05T05:42:54.116 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:54.116 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:42:54.508 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cepht:2202: flush_pg_stats: seq=171798691895 2024-02-05T05:42:54.508 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691895 2024-02-05T05:42:54.508 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673058 2-64424509532 3-304942678044 4-107374182476 5-455266533379 6-150323855421 7-171798691895' 2024-02-05T05:42:54.508 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:54.509 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:42:54.509 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528366 2024-02-05T05:42:54.509 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528366 2024-02-05T05:42:54.509 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673058 2-64424509532 3-304942678044 4-107374182476 5-455266533379 6-150323855421 7-171798691895 8-193273528366' 2024-02-05T05:42:54.509 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:54.509 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:42:54.510 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631492 2024-02-05T05:42:54.510 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631492 2024-02-05T05:42:54.510 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673058 2-64424509532 3-304942678044 4-107374182476 5-455266533379 6-150323855421 7-171798691895 8-193273528366 9-442381631492' 2024-02-05T05:42:54.510 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:54.510 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836587 2024-02-05T05:42:54.510 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:54.511 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:42:54.511 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836587 2024-02-05T05:42:54.511 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:54.511 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836587 2024-02-05T05:42:54.511 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-02-05T05:42:54.511 INFO:tasks.workunit.client.0.smithi148.stderr:224574: waiting osd.0 seq 21474836587 2024-02-05T05:42:54.512 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:54.512 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836587 -lt 21474836587 2024-02-05T05:42:54.512 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:54.512 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:54.512 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673058 2024-02-05T05:42:54.512 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:42:54.512 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673058 2024-02-05T05:42:54.513 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:54.513 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673058 2024-02-05T05:42:54.513 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-02-05T05:42:54.513 INFO:tasks.workunit.client.0.smithi148.stderr:224574: waiting osd.1 seq 42949673058 2024-02-05T05:42:55.110 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221est/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509530 2024-02-05T05:42:55.110 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:55.110 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509530 2024-02-05T05:42:55.110 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-02-05T05:42:55.110 INFO:tasks.workunit.client.0.smithi148.stderr:224575: waiting osd.2 seq 64424509530 2024-02-05T05:42:55.110 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:42:55.110 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509532 -lt 64424509530 2024-02-05T05:42:55.110 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:55.110 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678043 2024-02-05T05:42:55.110 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:55.110 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:42:55.111 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678043 2024-02-05T05:42:55.111 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:55.111 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678043 2024-02-05T05:42:55.111 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678043' 2024-02-05T05:42:55.111 INFO:tasks.workunit.client.0.smithi148.stderr:224575: waiting osd.3 seq 304942678043 2024-02-05T05:42:55.111 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:42:55.111 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678045 -lt 304942678043 2024-02-05T05:42:55.111 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:55.111 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182474 2024-02-05T05:42:55.111 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:55.111 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:42:55.111 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182474 2024-02-05T05:42:55.112 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:55.112 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182474 2024-02-05T05:42:55.112 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-02-05T05:42:55.112 INFO:tasks.workunit.client.0.smithi148.stderr:224575: waiting osd.4 seq 107374182474 2024-02-05T05:42:55.112 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:42:55.112 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182476 -lt 107374182474 2024-02-05T05:42:55.112 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:55.112 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-455266533378 2024-02-05T05:42:55.112 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:55.112 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:42:55.112 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-455266533378 2024-02-05T05:42:55.112 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:55.112 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533378 2024-02-05T05:42:55.783 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: 4: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:42:55.783 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673059 -lt 42949673058 2024-02-05T05:42:55.783 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:55.783 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509532 2024-02-05T05:42:55.783 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:55.783 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:42:55.783 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509532 2024-02-05T05:42:55.784 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:55.784 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509532 2024-02-05T05:42:55.784 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-02-05T05:42:55.784 INFO:tasks.workunit.client.0.smithi148.stderr:224574: waiting osd.2 seq 64424509532 2024-02-05T05:42:55.784 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:42:55.784 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509532 -lt 64424509532 2024-02-05T05:42:55.784 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:55.784 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678044 2024-02-05T05:42:55.784 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:55.784 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:42:55.784 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678044 2024-02-05T05:42:55.784 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:55.784 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678044 2024-02-05T05:42:55.785 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678044' 2024-02-05T05:42:55.785 INFO:tasks.workunit.client.0.smithi148.stderr:224574: waiting osd.3 seq 304942678044 2024-02-05T05:42:55.785 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:42:55.785 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678045 -lt 304942678044 2024-02-05T05:42:55.785 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:55.785 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182476 2024-02-05T05:42:55.785 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:55.785 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:42:55.785 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:55.785 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182476 2024-02-05T05:42:55.785 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182476 2024-02-05T05:42:55.785 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182476' 2024-02-05T05:42:55.785 INFO:tasks.workunit.client.0.smithi148.stderr:224574: waiting osd.4 seq 107374182476 2024-02-05T05:42:55.786 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:42:55.786 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182476 -lt 107374182476 2024-02-05T05:42:55.786 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:55.786 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-455266533379 2024-02-05T05:42:56.419 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //hecho 'waiting osd.5 seq 455266533378' 2024-02-05T05:42:56.419 INFO:tasks.workunit.client.0.smithi148.stderr:224575: waiting osd.5 seq 455266533378 2024-02-05T05:42:56.419 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:42:56.419 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533379 -lt 455266533378 2024-02-05T05:42:56.419 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:56.419 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855419 2024-02-05T05:42:56.419 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:56.419 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:42:56.419 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855419 2024-02-05T05:42:56.420 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:56.420 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855419 2024-02-05T05:42:56.420 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855419' 2024-02-05T05:42:56.420 INFO:tasks.workunit.client.0.smithi148.stderr:224575: waiting osd.6 seq 150323855419 2024-02-05T05:42:56.420 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:42:56.420 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855421 -lt 150323855419 2024-02-05T05:42:56.420 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:56.420 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691894 2024-02-05T05:42:56.420 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:56.420 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:42:56.420 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691894 2024-02-05T05:42:56.420 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:56.420 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691894 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691894' 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: waiting osd.7 seq 171798691894 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691895 -lt 171798691894 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528365 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528365 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528365 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: waiting osd.8 seq 193273528365 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:42:56.421 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528366 -lt 193273528365 2024-02-05T05:42:56.758 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221ome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:56.758 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:42:56.758 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-455266533379 2024-02-05T05:42:56.758 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533379 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 455266533379' 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: waiting osd.5 seq 455266533379 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533379 -lt 455266533379 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855421 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855421 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855421 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855421' 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: waiting osd.6 seq 150323855421 2024-02-05T05:42:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:42:56.760 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855421 -lt 150323855421 2024-02-05T05:42:56.760 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:56.760 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691895 2024-02-05T05:42:56.760 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:56.760 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:42:56.760 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691895 2024-02-05T05:42:56.760 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:56.760 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691895 2024-02-05T05:42:56.760 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691895' 2024-02-05T05:42:56.760 INFO:tasks.workunit.client.0.smithi148.stderr:224574: waiting osd.7 seq 171798691895 2024-02-05T05:42:56.760 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:42:56.760 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691896 -lt 171798691895 2024-02-05T05:42:56.760 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:56.760 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528366 2024-02-05T05:42:56.761 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:56.761 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:42:56.761 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528366 2024-02-05T05:42:57.906 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush0: flush_pg_stats: for s in $seqs 2024-02-05T05:42:57.906 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631491 2024-02-05T05:42:57.906 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:57.906 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:42:57.906 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631491 2024-02-05T05:42:57.906 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:57.906 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631491 2024-02-05T05:42:57.906 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631491' 2024-02-05T05:42:57.906 INFO:tasks.workunit.client.0.smithi148.stderr:224575: waiting osd.9 seq 442381631491 2024-02-05T05:42:57.906 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:42:57.906 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631492 -lt 442381631491 2024-02-05T05:42:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:42:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:42:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:42:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:42:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:42:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:42:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:42:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:42:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:42:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:42:57.907 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:42:57.908 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:57.908 INFO:tasks.workunit.client.0.smithi148.stderr:224575: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:57.908 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:42:57.908 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:42:57.908 INFO:tasks.workunit.client.0.smithi148.stderr:224575: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:42:57.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:42:58.560 INFO:tasks.workunit.client.0.smithi148.stderr:_pg_stats: cut -d - -f 2 2024-02-05T05:42:58.560 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528366 2024-02-05T05:42:58.560 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528366' 2024-02-05T05:42:58.561 INFO:tasks.workunit.client.0.smithi148.stderr:224574: waiting osd.8 seq 193273528366 2024-02-05T05:42:58.561 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:42:58.561 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528366 -lt 193273528366 2024-02-05T05:42:58.561 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:58.561 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631492 2024-02-05T05:42:58.561 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:58.561 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:42:58.561 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631492 2024-02-05T05:42:58.561 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:58.561 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631492 2024-02-05T05:42:58.561 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631492' 2024-02-05T05:42:58.561 INFO:tasks.workunit.client.0.smithi148.stderr:224574: waiting osd.9 seq 442381631492 2024-02-05T05:42:58.561 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:42:58.561 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631492 -lt 442381631492 2024-02-05T05:42:58.562 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:42:58.562 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:58.562 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:58.562 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:42:58.562 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:42:58.562 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:42:58.562 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:42:58.562 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:42:58.562 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:42:58.562 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:42:58.562 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:42:58.562 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:42:58.562 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:42:58.563 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:58.563 INFO:tasks.workunit.client.0.smithi148.stderr:224574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:58.563 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:42:58.563 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:42:58.563 INFO:tasks.workunit.client.0.smithi148.stderr:224574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:42:58.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:42:58.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:42:58.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 224571 2024-02-05T05:42:58.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:42:58.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:42:58.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:42:58.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:42:58.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:42:58.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:42:58.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:42:58.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-02-05T05:42:58.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:42:58.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:42:58.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-02-05T05:42:58.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-02-05T05:42:58.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:42:58.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:42:58.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:42:58.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:42:58.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 227827"' 2024-02-05T05:42:58.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 227827' 2024-02-05T05:42:58.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:42:58.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:42:58.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:42:58.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:42:58.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 227829"' 2024-02-05T05:42:58.604 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 227829' 2024-02-05T05:42:58.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:42:58.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/227830: /' 2024-02-05T05:42:58.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 227827 227829' 2024-02-05T05:42:58.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:42:58.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:42:58.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 227827 2024-02-05T05:42:58.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:42:58.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/227832: /' 2024-02-05T05:43:01.800 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:43:01.801 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:43:01.801 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:43:01.801 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:43:01.801 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:43:01.801 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:43:01.801 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:43:01.801 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:43:01.801 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:43:01.801 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:43:01.801 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:43:01.801 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:43:01.801 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:43:01.802 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:43:01.802 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:43:01.802 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:43:01.802 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:43:01.802 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:43:01.802 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:43:01.802 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:43:01.802 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:43:01.802 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:43:01.803 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-02-05T05:43:01.803 INFO:tasks.workunit.client.0.smithi148.stderr:227832: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:43:01.803 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:43:01.803 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:43:01.803 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:43:01.803 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:43:01.803 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:43:01.803 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:43:01.803 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:43:01.803 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:43:01.806 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:43:01.807 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:43:01.807 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:43:01.807 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:43:01.807 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:43:01.807 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:43:01.807 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:43:01.807 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:43:01.807 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:43:01.807 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:43:01.807 INFO:tasks.workunit.client.0.smithi148.stderr:227832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:43:01.807 INFO:tasks.workunit.client.0.smithi148.stderr:227832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:01.808 INFO:tasks.workunit.client.0.smithi148.stderr:227832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:43:01.808 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:43:01.808 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:43:01.808 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:43:01.808 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:43:01.808 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:43:01.808 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:43:01.808 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:43:01.808 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:43:01.808 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:43:01.808 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:43:01.808 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:43:01.808 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:43:01.809 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:43:01.809 INFO:tasks.workunit.client.0.smithi148.stderr:227832: start osd.5 2024-02-05T05:43:01.809 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:43:01.905 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:43:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:43:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:43:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:43:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:43:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:43:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:43:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:43:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:43:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:43:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:43:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:43:01.906 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:43:01.907 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:43:01.907 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:43:01.907 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:43:01.907 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:43:01.907 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:43:01.907 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:43:01.907 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:43:01.907 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:43:01.907 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:43:01.907 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:43:01.907 INFO:tasks.workunit.client.0.smithi148.stderr:227830: remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:43:01.907 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:43:01.907 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:43:01.907 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:43:01.908 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:43:01.908 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:43:01.908 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:43:01.908 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:43:01.908 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:43:01.910 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:43:01.910 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:43:01.910 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:43:01.910 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:43:01.910 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:43:01.910 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:43:01.910 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:43:01.911 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:43:01.911 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:43:01.911 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:43:01.911 INFO:tasks.workunit.client.0.smithi148.stderr:227830: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:43:01.911 INFO:tasks.workunit.client.0.smithi148.stderr:227830: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:01.911 INFO:tasks.workunit.client.0.smithi148.stderr:227830: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:43:01.911 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:43:01.911 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:43:01.911 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:43:01.911 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:43:01.911 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:43:01.911 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:43:01.912 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:43:01.912 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:43:01.912 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:43:01.912 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:43:01.912 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:43:01.912 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:43:01.912 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:43:01.912 INFO:tasks.workunit.client.0.smithi148.stderr:227830: start osd.3 2024-02-05T05:43:01.912 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:43:06.554 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:43:06.554 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:43:06.554 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:43:06.554 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:43:06.554 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:43:06.554 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 2024-02-05T05:43:01.838+0000 7fe30b457640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:06.554 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 2024-02-05T05:43:01.842+0000 7fe30b457640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:06.554 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 2024-02-05T05:43:01.842+0000 7fe30b457640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:06.554 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:43:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:43:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:43:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:43:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:43:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:43:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 0 2024-02-05T05:43:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:43:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 2024-02-05T05:43:02.398+0000 7fe30b457640 -1 Falling back to public interface 2024-02-05T05:43:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:06.555 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:43:06.556 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 1 2024-02-05T05:43:06.556 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:06.556 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:43:06.556 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:06.556 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:06.556 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:06.556 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:43:06.556 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 2 2024-02-05T05:43:06.556 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:06.556 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:43:06.556 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:06.556 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 2024-02-05T05:43:05.338+0000 7fe30b457640 -1 osd.5 108 log_to_monitors true 2024-02-05T05:43:06.556 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:06.556 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:06.557 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:43:06.557 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 3 2024-02-05T05:43:06.557 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:06.557 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:43:06.557 INFO:tasks.workunit.client.0.smithi148.stderr:227832: osd.5 up in weight 1 up_from 113 up_thru 106 down_at 109 l227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:43:06.557 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:43:06.557 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:43:06.557 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:43:06.557 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:43:06.558 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 2024-02-05T05:43:01.938+0000 7fa29f130640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:06.558 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 2024-02-05T05:43:01.942+0000 7fa29f130640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:06.558 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 2024-02-05T05:43:01.942+0000 7fa29f130640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:06.558 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:43:06.558 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:43:06.558 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:43:06.558 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:43:06.558 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:43:06.558 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:06.558 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:43:06.558 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 0 2024-02-05T05:43:06.558 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:06.558 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:43:06.559 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 2024-02-05T05:43:02.490+0000 7fa29f130640 -1 Falling back to public interface 2024-02-05T05:43:06.559 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:06.559 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:06.559 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:06.559 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:43:06.559 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 1 2024-02-05T05:43:06.559 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:06.559 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:43:06.559 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:06.559 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:06.559 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:06.559 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:43:06.559 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 2 2024-02-05T05:43:06.559 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:06.560 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:43:06.560 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:06.560 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 2024-02-05T05:43:05.554+0000 7fa29f130640 -1 osd.3 108 log_to_monitors true 2024-02-05T05:43:06.560 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:06.560 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:06.560 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:43:06.560 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 3 2024-02-05T05:43:06.560 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:06.560 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:43:07.845 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeast_clean_interval [106,108) [v2:127.0.0.1:6826/1107067372,v1:127.0.0.1:6827/1107067372] [v2:127.0.0.1:6828/1107067372,v1:127.0.0.1:6829/1107067372] exists,up f12201fc-c63f-4591-a81c-5c4646aaab07 2024-02-05T05:43:07.846 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:43:07.846 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:43:07.846 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:43:07.846 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:43:07.846 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:43:07.846 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:43:07.846 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:43:07.846 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:43:07.846 INFO:tasks.workunit.client.0.smithi148.stderr:227832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:43:07.846 INFO:tasks.workunit.client.0.smithi148.stderr:227832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:43:07.846 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:43:07.846 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 1 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 2 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 3 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 4 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 5 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 6 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 7 2024-02-05T05:43:07.847 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 8 2024-02-05T05:43:07.848 INFO:tasks.workunit.client.0.smithi148.stderr:227832: 9' 2024-02-05T05:43:07.848 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:43:07.848 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:07.848 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:43:07.848 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836591 2024-02-05T05:43:07.848 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836591 2024-02-05T05:43:07.848 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591' 2024-02-05T05:43:07.848 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:07.848 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:43:07.848 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673062 2024-02-05T05:43:07.848 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673062 2024-02-05T05:43:07.848 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673062' 2024-02-05T05:43:07.848 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:08.945 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.cliers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:08.945 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:08.946 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:08.946 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:43:08.946 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 4 2024-02-05T05:43:08.946 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:08.946 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:43:08.946 INFO:tasks.workunit.client.0.smithi148.stderr:227830: osd.3 up in weight 1 up_from 114 up_thru 114 down_at 109 last_clean_interval [71,108) [v2:127.0.0.1:6874/331562525,v1:127.0.0.1:6875/331562525] [v2:127.0.0.1:6876/331562525,v1:127.0.0.1:6877/331562525] exists,up b0f217d6-0908-40e7-b61a-8971d53ffef2 2024-02-05T05:43:08.946 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:43:08.946 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:43:08.946 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:43:08.946 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:43:08.946 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:43:08.946 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:43:08.946 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:43:08.947 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:43:08.947 INFO:tasks.workunit.client.0.smithi148.stderr:227830: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:43:08.947 INFO:tasks.workunit.client.0.smithi148.stderr:227830: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:43:08.947 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:43:08.947 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:43:08.947 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:43:08.947 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:43:08.947 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:43:08.947 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:43:08.947 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:43:08.947 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:43:08.947 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:43:08.947 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:43:08.947 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 1 2024-02-05T05:43:08.948 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 2 2024-02-05T05:43:08.948 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 3 2024-02-05T05:43:08.948 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 4 2024-02-05T05:43:08.948 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 5 2024-02-05T05:43:08.948 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 6 2024-02-05T05:43:08.948 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 7 2024-02-05T05:43:08.948 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 8 2024-02-05T05:43:08.948 INFO:tasks.workunit.client.0.smithi148.stderr:227830: 9' 2024-02-05T05:43:08.948 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:43:08.948 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:08.948 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:43:08.948 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836592 2024-02-05T05:43:08.948 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836592 2024-02-05T05:43:08.949 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592' 2024-02-05T05:43:08.949 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:09.571 INFO:tasks.workunit.client.0.smithi148.stderr:227830:nt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:43:09.571 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509536 2024-02-05T05:43:09.571 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509536 2024-02-05T05:43:09.572 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673062 2-64424509536' 2024-02-05T05:43:09.572 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:09.572 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:43:09.572 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=489626271746 2024-02-05T05:43:09.572 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 489626271746 2024-02-05T05:43:09.572 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673062 2-64424509536 3-489626271746' 2024-02-05T05:43:09.572 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:09.572 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:43:09.572 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182480 2024-02-05T05:43:09.572 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182480 2024-02-05T05:43:09.572 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673062 2-64424509536 3-489626271746 4-107374182480' 2024-02-05T05:43:09.572 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:09.573 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:43:09.573 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304450 2024-02-05T05:43:09.573 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304450 2024-02-05T05:43:09.573 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673062 2-64424509536 3-489626271746 4-107374182480 5-485331304450' 2024-02-05T05:43:09.573 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:09.573 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:43:09.573 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855425 2024-02-05T05:43:09.573 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855425 2024-02-05T05:43:09.573 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673062 2-64424509536 3-489626271746 4-107374182480 5-485331304450 6-150323855425' 2024-02-05T05:43:09.573 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:09.573 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:43:09.573 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691899 2024-02-05T05:43:09.573 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691899 2024-02-05T05:43:09.573 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673062 2-64424509536 3-489626271746 4-107374182480 5-485331304450 6-150323855425 7-171798691899' 2024-02-05T05:43:09.574 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:09.574 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:43:10.124 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtes //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:43:10.125 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673064 2024-02-05T05:43:10.125 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673064 2024-02-05T05:43:10.125 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673064' 2024-02-05T05:43:10.125 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:10.125 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:43:10.125 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509537 2024-02-05T05:43:10.125 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509537 2024-02-05T05:43:10.125 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673064 2-64424509537' 2024-02-05T05:43:10.125 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:10.125 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:43:10.125 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=489626271747 2024-02-05T05:43:10.125 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 489626271747 2024-02-05T05:43:10.125 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673064 2-64424509537 3-489626271747' 2024-02-05T05:43:10.126 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:10.126 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:43:10.126 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182481 2024-02-05T05:43:10.126 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182481 2024-02-05T05:43:10.126 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673064 2-64424509537 3-489626271747 4-107374182481' 2024-02-05T05:43:10.126 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:10.126 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:43:10.126 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304451 2024-02-05T05:43:10.126 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304451 2024-02-05T05:43:10.126 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673064 2-64424509537 3-489626271747 4-107374182481 5-485331304451' 2024-02-05T05:43:10.126 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:10.126 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:43:10.126 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855426 2024-02-05T05:43:10.126 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855426 2024-02-05T05:43:10.127 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673064 2-64424509537 3-489626271747 4-107374182481 5-485331304451 6-150323855426' 2024-02-05T05:43:10.127 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:10.127 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:43:10.372 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flust/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528370 2024-02-05T05:43:10.372 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528370 2024-02-05T05:43:10.372 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673062 2-64424509536 3-489626271746 4-107374182480 5-485331304450 6-150323855425 7-171798691899 8-193273528370' 2024-02-05T05:43:10.373 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:10.373 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:43:10.373 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631496 2024-02-05T05:43:10.373 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631496 2024-02-05T05:43:10.373 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673062 2-64424509536 3-489626271746 4-107374182480 5-485331304450 6-150323855425 7-171798691899 8-193273528370 9-442381631496' 2024-02-05T05:43:10.373 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:10.373 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836591 2024-02-05T05:43:10.373 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:10.373 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:43:10.373 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836591 2024-02-05T05:43:10.373 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:10.373 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836591 2024-02-05T05:43:10.373 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-02-05T05:43:10.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: waiting osd.0 seq 21474836591 2024-02-05T05:43:10.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:10.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836591 -lt 21474836591 2024-02-05T05:43:10.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:10.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673062 2024-02-05T05:43:10.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:10.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:43:10.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673062 2024-02-05T05:43:10.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:10.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673062 2024-02-05T05:43:10.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-02-05T05:43:10.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: waiting osd.1 seq 42949673062 2024-02-05T05:43:10.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:43:10.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673064 -lt 42949673062 2024-02-05T05:43:10.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:10.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509536 2024-02-05T05:43:10.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:10.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:43:10.767 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/h_pg_stats: seq=171798691900 2024-02-05T05:43:10.768 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691900 2024-02-05T05:43:10.768 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673064 2-64424509537 3-489626271747 4-107374182481 5-485331304451 6-150323855426 7-171798691900' 2024-02-05T05:43:10.768 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:10.768 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:43:10.768 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528371 2024-02-05T05:43:10.768 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528371 2024-02-05T05:43:10.768 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673064 2-64424509537 3-489626271747 4-107374182481 5-485331304451 6-150323855426 7-171798691900 8-193273528371' 2024-02-05T05:43:10.768 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:10.768 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:43:10.768 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631497 2024-02-05T05:43:10.768 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631497 2024-02-05T05:43:10.768 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673064 2-64424509537 3-489626271747 4-107374182481 5-485331304451 6-150323855426 7-171798691900 8-193273528371 9-442381631497' 2024-02-05T05:43:10.768 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:10.768 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836592 2024-02-05T05:43:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:43:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836592 2024-02-05T05:43:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836592 2024-02-05T05:43:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2024-02-05T05:43:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:227830: waiting osd.0 seq 21474836592 2024-02-05T05:43:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836592 -lt 21474836592 2024-02-05T05:43:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673064 2024-02-05T05:43:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:43:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673064 2024-02-05T05:43:10.770 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:10.770 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673064 2024-02-05T05:43:10.770 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673064' 2024-02-05T05:43:10.770 INFO:tasks.workunit.client.0.smithi148.stderr:227830: waiting osd.1 seq 42949673064 2024-02-05T05:43:11.373 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pgclone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509536 2024-02-05T05:43:11.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:11.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509536 2024-02-05T05:43:11.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-02-05T05:43:11.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: waiting osd.2 seq 64424509536 2024-02-05T05:43:11.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:43:11.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509536 2024-02-05T05:43:11.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:11.374 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-489626271746 2024-02-05T05:43:11.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:11.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:43:11.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-489626271746 2024-02-05T05:43:11.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:11.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=489626271746 2024-02-05T05:43:11.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 489626271746' 2024-02-05T05:43:11.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: waiting osd.3 seq 489626271746 2024-02-05T05:43:11.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:43:11.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271747 -lt 489626271746 2024-02-05T05:43:11.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:11.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182480 2024-02-05T05:43:11.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:11.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:43:11.375 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182480 2024-02-05T05:43:11.376 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:11.376 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182480 2024-02-05T05:43:11.376 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182480' 2024-02-05T05:43:11.376 INFO:tasks.workunit.client.0.smithi148.stderr:227832: waiting osd.4 seq 107374182480 2024-02-05T05:43:11.376 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:43:11.376 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182481 -lt 107374182480 2024-02-05T05:43:11.376 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:11.376 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:11.376 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-485331304450 2024-02-05T05:43:11.376 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:43:11.376 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-485331304450 2024-02-05T05:43:11.376 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:11.376 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304450 2024-02-05T05:43:12.077 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo_stats: ceph osd last-stat-seq 1 2024-02-05T05:43:12.077 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673064 -lt 42949673064 2024-02-05T05:43:12.077 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:12.077 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509537 2024-02-05T05:43:12.077 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:12.077 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:43:12.077 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509537 2024-02-05T05:43:12.077 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:12.077 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509537 2024-02-05T05:43:12.077 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-02-05T05:43:12.078 INFO:tasks.workunit.client.0.smithi148.stderr:227830: waiting osd.2 seq 64424509537 2024-02-05T05:43:12.078 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:43:12.078 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509537 2024-02-05T05:43:12.078 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:12.078 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-489626271747 2024-02-05T05:43:12.078 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:12.078 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:43:12.078 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-489626271747 2024-02-05T05:43:12.078 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:12.078 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=489626271747 2024-02-05T05:43:12.078 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 489626271747' 2024-02-05T05:43:12.078 INFO:tasks.workunit.client.0.smithi148.stderr:227830: waiting osd.3 seq 489626271747 2024-02-05T05:43:12.078 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:43:12.079 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271747 -lt 489626271747 2024-02-05T05:43:12.079 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:12.079 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182481 2024-02-05T05:43:12.079 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:12.079 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:43:12.079 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182481 2024-02-05T05:43:12.079 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:12.079 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182481 2024-02-05T05:43:12.079 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-02-05T05:43:12.079 INFO:tasks.workunit.client.0.smithi148.stderr:227830: waiting osd.4 seq 107374182481 2024-02-05T05:43:12.079 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:43:12.079 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182481 -lt 107374182481 2024-02-05T05:43:12.079 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:12.079 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:12.647 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephte 'waiting osd.5 seq 485331304450' 2024-02-05T05:43:12.647 INFO:tasks.workunit.client.0.smithi148.stderr:227832: waiting osd.5 seq 485331304450 2024-02-05T05:43:12.647 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:43:12.647 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304451 -lt 485331304450 2024-02-05T05:43:12.647 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:12.647 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855425 2024-02-05T05:43:12.647 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:12.647 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:43:12.647 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855425 2024-02-05T05:43:12.647 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:12.647 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855425 2024-02-05T05:43:12.647 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855425' 2024-02-05T05:43:12.648 INFO:tasks.workunit.client.0.smithi148.stderr:227832: waiting osd.6 seq 150323855425 2024-02-05T05:43:12.648 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:43:12.648 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855426 -lt 150323855425 2024-02-05T05:43:12.648 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:12.648 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:12.648 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691899 2024-02-05T05:43:12.648 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:43:12.648 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691899 2024-02-05T05:43:12.648 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:12.648 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691899 2024-02-05T05:43:12.648 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691899' 2024-02-05T05:43:12.648 INFO:tasks.workunit.client.0.smithi148.stderr:227832: waiting osd.7 seq 171798691899 2024-02-05T05:43:12.648 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:43:12.648 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691901 -lt 171798691899 2024-02-05T05:43:12.649 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:12.649 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:12.649 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528370 2024-02-05T05:43:12.649 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:43:12.649 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528370 2024-02-05T05:43:12.649 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:12.649 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528370 2024-02-05T05:43:12.649 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528370' 2024-02-05T05:43:12.649 INFO:tasks.workunit.client.0.smithi148.stderr:227832: waiting osd.8 seq 193273528370 2024-02-05T05:43:12.649 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:43:12.649 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528372 -lt 193273528370 2024-02-05T05:43:13.212 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fst/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-485331304451 2024-02-05T05:43:13.212 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:43:13.212 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-485331304451 2024-02-05T05:43:13.212 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:13.212 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304451 2024-02-05T05:43:13.212 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 485331304451' 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: waiting osd.5 seq 485331304451 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304452 -lt 485331304451 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855426 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855426 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855426 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855426' 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: waiting osd.6 seq 150323855426 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855426 -lt 150323855426 2024-02-05T05:43:13.213 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691900 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691900 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691900 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: waiting osd.7 seq 171798691900 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691901 -lt 171798691900 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528371 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:43:13.214 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528371 2024-02-05T05:43:14.366 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: lush_pg_stats: for s in $seqs 2024-02-05T05:43:14.366 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631496 2024-02-05T05:43:14.367 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:14.367 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:43:14.367 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631496 2024-02-05T05:43:14.367 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:14.367 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631496 2024-02-05T05:43:14.367 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631496' 2024-02-05T05:43:14.367 INFO:tasks.workunit.client.0.smithi148.stderr:227832: waiting osd.9 seq 442381631496 2024-02-05T05:43:14.367 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:43:14.367 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631497 -lt 442381631496 2024-02-05T05:43:14.367 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:43:14.367 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:14.367 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:14.367 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:43:14.367 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:14.368 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:14.368 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:14.368 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:14.368 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:14.368 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:14.368 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:14.368 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:43:14.368 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:14.368 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:14.368 INFO:tasks.workunit.client.0.smithi148.stderr:227832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:14.368 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:43:14.368 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:43:14.369 INFO:tasks.workunit.client.0.smithi148.stderr:227832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:43:14.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:43:14.983 INFO:tasks.workunit.client.0.smithi148.stderr: cut -d - -f 2 2024-02-05T05:43:14.983 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528371 2024-02-05T05:43:14.983 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528371' 2024-02-05T05:43:14.983 INFO:tasks.workunit.client.0.smithi148.stderr:227830: waiting osd.8 seq 193273528371 2024-02-05T05:43:14.983 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:43:14.983 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528372 -lt 193273528371 2024-02-05T05:43:14.983 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:14.983 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631497 2024-02-05T05:43:14.983 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:14.983 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:43:14.983 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631497 2024-02-05T05:43:14.984 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:14.984 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631497 2024-02-05T05:43:14.984 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631497' 2024-02-05T05:43:14.984 INFO:tasks.workunit.client.0.smithi148.stderr:227830: waiting osd.9 seq 442381631497 2024-02-05T05:43:14.984 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:43:14.984 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631497 -lt 442381631497 2024-02-05T05:43:14.984 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:43:14.984 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:14.984 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:14.984 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:43:14.984 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:14.984 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:14.984 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:14.985 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:14.985 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:14.985 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:14.985 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:14.985 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:43:14.985 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:14.985 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:14.985 INFO:tasks.workunit.client.0.smithi148.stderr:227830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:14.985 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:43:14.985 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:43:14.985 INFO:tasks.workunit.client.0.smithi148.stderr:227830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:43:14.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:43:14.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:43:14.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 227829 2024-02-05T05:43:14.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:43:14.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:43:14.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:43:14.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:43:14.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:43:14.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:43:14.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:43:14.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:43:14.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:43:14.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:43:15.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:43:15.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:43:15.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:43:15.315 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:43:15.315 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:43:15.315 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:43:15.316 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:43:15.316 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:43:15.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:42:37.329889+0000 2024-02-05T05:43:15.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:43:15.918 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:43:15.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:42:37.329889+0000 2024-02-05T05:43:15.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:43:15.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:42:37.329889+0000 2024-02-05T05:43:15.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:43:15.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:43:15.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:43:15.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:43:15.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:43:15.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:43:15.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:43:15.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:43:16.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:42:37.329889+0000 '>' 2024-02-05T05:42:37.329889+0000 2024-02-05T05:43:16.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:43:17.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:43:17.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:43:17.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:43:17.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:43:17.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:43:17.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:43:17.241 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:43:17.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:42:37.329889+0000 '>' 2024-02-05T05:42:37.329889+0000 2024-02-05T05:43:17.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:43:18.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:43:18.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:43:18.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:43:18.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:43:18.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:43:18.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:43:18.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:43:18.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:42:37.329889+0000 '>' 2024-02-05T05:42:37.329889+0000 2024-02-05T05:43:18.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:43:19.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:43:19.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:43:19.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:43:19.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:43:19.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:43:19.859 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:43:19.859 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:43:20.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:42:37.329889+0000 '>' 2024-02-05T05:42:37.329889+0000 2024-02-05T05:43:20.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:43:21.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:43:21.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:43:21.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:43:21.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:43:21.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:43:21.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:43:21.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:43:21.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:42:37.329889+0000 '>' 2024-02-05T05:42:37.329889+0000 2024-02-05T05:43:21.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:43:22.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:43:22.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:43:22.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:43:22.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:43:22.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:43:22.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:43:22.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:43:22.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:43:16.759682+0000 '>' 2024-02-05T05:42:37.329889+0000 2024-02-05T05:43:22.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:43:22.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:43:22.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:43:22.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:43:22.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:43:22.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 231283"' 2024-02-05T05:43:22.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 231283' 2024-02-05T05:43:22.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:43:22.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:43:22.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:43:22.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 231285"' 2024-02-05T05:43:22.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:43:22.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 231285' 2024-02-05T05:43:22.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:43:22.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 231283 231285' 2024-02-05T05:43:22.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:43:22.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:43:22.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/231286: /' 2024-02-05T05:43:22.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 231283 2024-02-05T05:43:22.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:43:22.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/231288: /' 2024-02-05T05:43:26.903 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:43:26.903 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:43:26.903 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:43:26.904 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:43:26.904 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:43:26.904 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:43:26.904 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:43:26.904 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:43:26.904 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:43:26.904 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:43:26.904 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:43:26.904 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:43:26.904 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:43:26.904 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:43:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:43:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:43:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:43:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:43:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:43:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:43:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:43:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:43:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:43:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:231286: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:43:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:231286: _ 2024-02-05T05:43:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:231286: hinfo_key 2024-02-05T05:43:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:231286: snapset 2024-02-05T05:43:26.906 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:43:26.906 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:43:26.906 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:43:26.906 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:43:26.906 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:43:26.906 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:43:26.906 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:43:26.908 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:43:26.908 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:43:26.908 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:43:26.908 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:43:26.908 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:43:26.908 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:43:26.908 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:43:26.909 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:43:26.909 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:43:26.909 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:43:26.909 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:43:26.909 INFO:tasks.workunit.client.0.smithi148.stderr:231286: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:43:26.909 INFO:tasks.workunit.client.0.smithi148.stderr:231286: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:26.909 INFO:tasks.workunit.client.0.smithi148.stderr:231286: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:43:26.909 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:43:26.909 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:43:26.909 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:43:26.909 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:43:26.909 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:43:26.909 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:43:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:43:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:43:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:43:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:43:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:43:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:43:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:43:26.910 INFO:tasks.workunit.client.0.smithi148.stderr:231286: start osd.3 2024-02-05T05:43:27.450 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:43:27.450 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:43:27.450 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:43:27.450 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:43:27.450 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:43:27.450 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:43:27.451 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:43:27.451 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:43:27.451 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:43:27.451 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:43:27.451 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:43:27.451 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:43:27.451 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:43:27.451 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:43:27.451 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:43:27.451 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:43:27.451 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:43:27.451 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:43:27.452 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:43:27.452 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:43:27.452 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:43:27.452 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:43:27.452 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-02-05T05:43:27.452 INFO:tasks.workunit.client.0.smithi148.stderr:231288: _ 2024-02-05T05:43:27.452 INFO:tasks.workunit.client.0.smithi148.stderr:231288: hinfo_key 2024-02-05T05:43:27.452 INFO:tasks.workunit.client.0.smithi148.stderr:231288: snapset 2024-02-05T05:43:27.452 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:43:27.452 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:43:27.452 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:43:27.453 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:43:27.453 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:43:27.453 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:43:27.453 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:43:27.453 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:43:27.454 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:43:27.454 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:43:27.454 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:43:27.454 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:43:27.454 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:43:27.454 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:43:27.454 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:43:27.454 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:43:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:43:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:43:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:231288: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:43:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:231288: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:231288: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:43:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:43:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:43:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:43:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:43:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:43:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:43:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:43:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:43:27.456 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:43:27.456 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:43:27.456 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:43:27.456 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:43:27.456 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:43:27.456 INFO:tasks.workunit.client.0.smithi148.stderr:231288: start osd.5 2024-02-05T05:43:31.241 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:43:31.241 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:43:31.241 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:43:31.242 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:43:31.242 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:43:31.242 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:43:31.242 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 2024-02-05T05:43:26.946+0000 7f90d6c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:31.242 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 2024-02-05T05:43:26.946+0000 7f90d6c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:31.242 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 2024-02-05T05:43:26.950+0000 7f90d6c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:31.242 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:43:31.242 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:43:31.242 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:43:31.242 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:43:31.242 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:43:31.243 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:31.243 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:43:31.243 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 0 2024-02-05T05:43:31.243 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:31.243 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:43:31.243 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 2024-02-05T05:43:27.502+0000 7f90d6c0d640 -1 Falling back to public interface 2024-02-05T05:43:31.243 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:31.243 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:31.243 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:31.243 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:43:31.243 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 1 2024-02-05T05:43:31.244 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:31.244 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:43:31.244 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:31.244 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:31.244 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:31.244 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:43:31.244 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 2 2024-02-05T05:43:31.244 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:31.244 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:43:31.244 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:31.244 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 2024-02-05T05:43:30.282+0000 7f90d6c0d640 -1 osd.3 116 log_to_monitors true 2024-02-05T05:43:31.244 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:31.245 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:31.245 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:43:31.245 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 3 2024-02-05T05:43:31.245 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:32.444 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-02-05T05:43:32.444 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:43:32.444 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:43:32.444 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:43:32.444 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:43:32.444 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:43:32.444 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 2024-02-05T05:43:27.486+0000 7f5b4bce8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 2024-02-05T05:43:27.490+0000 7f5b4bce8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 2024-02-05T05:43:27.490+0000 7f5b4bce8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:43:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:43:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:43:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:43:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:43:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:43:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 0 2024-02-05T05:43:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:43:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 2024-02-05T05:43:28.030+0000 7f5b4bce8640 -1 Falling back to public interface 2024-02-05T05:43:32.445 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:32.446 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:32.446 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:32.446 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:43:32.446 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 1 2024-02-05T05:43:32.446 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:32.446 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:43:32.446 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:32.446 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:32.446 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:32.446 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:43:32.446 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 2 2024-02-05T05:43:32.447 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:32.447 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:43:32.447 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:32.447 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 2024-02-05T05:43:30.930+0000 7f5b4bce8640 -1 osd.5 116 log_to_monitors true 2024-02-05T05:43:32.447 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:32.447 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:32.447 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:43:32.447 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 3 2024-02-05T05:43:32.447 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:32.447 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:43:33.212 INFO:tasks.workunit.client.0.smithi148.stderr:231288: osd.5 up in weight 1 up_from 122 up_thru 113 dowrs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:43:33.213 INFO:tasks.workunit.client.0.smithi148.stderr:231286: osd.3 up in weight 1 up_from 121 up_thru 114 down_at 117 last_clean_interval [114,116) [v2:127.0.0.1:6826/2260614165,v1:127.0.0.1:6827/2260614165] [v2:127.0.0.1:6828/2260614165,v1:127.0.0.1:6829/2260614165] exists,up b0f217d6-0908-40e7-b61a-8971d53ffef2 2024-02-05T05:43:33.213 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:43:33.213 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:43:33.213 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:43:33.214 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:43:33.214 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:43:33.214 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:43:33.214 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:43:33.214 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:43:33.214 INFO:tasks.workunit.client.0.smithi148.stderr:231286: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:43:33.215 INFO:tasks.workunit.client.0.smithi148.stderr:231286: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:43:33.215 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:43:33.215 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:43:33.215 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:43:33.215 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:43:33.215 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:43:33.216 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:43:33.216 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:43:33.216 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:43:33.216 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:43:33.216 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:43:33.217 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 1 2024-02-05T05:43:33.217 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 2 2024-02-05T05:43:33.217 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 3 2024-02-05T05:43:33.217 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 4 2024-02-05T05:43:33.217 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 5 2024-02-05T05:43:33.217 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 6 2024-02-05T05:43:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 7 2024-02-05T05:43:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 8 2024-02-05T05:43:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:231286: 9' 2024-02-05T05:43:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:43:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:33.218 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:43:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836598 2024-02-05T05:43:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836598 2024-02-05T05:43:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598' 2024-02-05T05:43:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:43:33.219 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673069 2024-02-05T05:43:33.220 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673069 2024-02-05T05:43:33.220 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673069' 2024-02-05T05:43:33.712 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0n_at 117 last_clean_interval [113,116) [v2:127.0.0.1:6874/2211255675,v1:127.0.0.1:6875/2211255675] [v2:127.0.0.1:6876/2211255675,v1:127.0.0.1:6877/2211255675] exists,up f12201fc-c63f-4591-a81c-5c4646aaab07 2024-02-05T05:43:33.712 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:43:33.712 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:43:33.712 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:43:33.713 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:43:33.713 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:43:33.713 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:43:33.713 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:43:33.713 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:43:33.713 INFO:tasks.workunit.client.0.smithi148.stderr:231288: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:43:33.713 INFO:tasks.workunit.client.0.smithi148.stderr:231288: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:43:33.713 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:43:33.713 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:43:33.713 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:43:33.713 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:43:33.714 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:43:33.714 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:43:33.714 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:43:33.714 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:43:33.714 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:43:33.714 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:43:33.714 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 1 2024-02-05T05:43:33.714 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 2 2024-02-05T05:43:33.714 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 3 2024-02-05T05:43:33.714 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 4 2024-02-05T05:43:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 5 2024-02-05T05:43:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 6 2024-02-05T05:43:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 7 2024-02-05T05:43:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 8 2024-02-05T05:43:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:231288: 9' 2024-02-05T05:43:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:43:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:43:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836599 2024-02-05T05:43:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836599 2024-02-05T05:43:33.716 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599' 2024-02-05T05:43:33.716 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:33.716 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:43:33.716 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673071 2024-02-05T05:43:33.716 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673071 2024-02-05T05:43:33.716 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673071' 2024-02-05T05:43:33.716 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:34.664 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest//qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:34.664 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:43:34.664 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509543 2024-02-05T05:43:34.664 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509543 2024-02-05T05:43:34.664 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673069 2-64424509543' 2024-02-05T05:43:34.664 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:34.664 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:43:34.664 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=519691042818 2024-02-05T05:43:34.664 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 519691042818 2024-02-05T05:43:34.665 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673069 2-64424509543 3-519691042818' 2024-02-05T05:43:34.665 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:34.665 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:43:34.665 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182487 2024-02-05T05:43:34.665 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182487 2024-02-05T05:43:34.665 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673069 2-64424509543 3-519691042818 4-107374182487' 2024-02-05T05:43:34.665 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:34.665 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:43:34.665 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010114 2024-02-05T05:43:34.665 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010114 2024-02-05T05:43:34.665 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673069 2-64424509543 3-519691042818 4-107374182487 5-523986010114' 2024-02-05T05:43:34.665 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:34.666 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:43:34.666 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855432 2024-02-05T05:43:34.666 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855432 2024-02-05T05:43:34.666 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673069 2-64424509543 3-519691042818 4-107374182487 5-523986010114 6-150323855432' 2024-02-05T05:43:34.666 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:34.666 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:43:34.666 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691906 2024-02-05T05:43:34.666 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691906 2024-02-05T05:43:34.666 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673069 2-64424509543 3-519691042818 4-107374182487 5-523986010114 6-150323855432 7-171798691906' 2024-02-05T05:43:34.666 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:35.128 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qclone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:43:35.128 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509544 2024-02-05T05:43:35.128 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509544 2024-02-05T05:43:35.128 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673071 2-64424509544' 2024-02-05T05:43:35.128 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:35.128 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:43:35.128 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=519691042819 2024-02-05T05:43:35.128 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 519691042819 2024-02-05T05:43:35.128 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673071 2-64424509544 3-519691042819' 2024-02-05T05:43:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:43:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182488 2024-02-05T05:43:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182488 2024-02-05T05:43:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673071 2-64424509544 3-519691042819 4-107374182488' 2024-02-05T05:43:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:43:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010115 2024-02-05T05:43:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010115 2024-02-05T05:43:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673071 2-64424509544 3-519691042819 4-107374182488 5-523986010115' 2024-02-05T05:43:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:43:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855433 2024-02-05T05:43:35.130 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855433 2024-02-05T05:43:35.130 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673071 2-64424509544 3-519691042819 4-107374182488 5-523986010115 6-150323855433' 2024-02-05T05:43:35.130 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:35.130 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:43:35.130 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691907 2024-02-05T05:43:35.130 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691907 2024-02-05T05:43:35.130 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673071 2-64424509544 3-519691042819 4-107374182488 5-523986010115 6-150323855433 7-171798691907' 2024-02-05T05:43:35.130 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:35.130 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:43:35.649 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuna/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:43:35.649 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528377 2024-02-05T05:43:35.649 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528377 2024-02-05T05:43:35.649 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673069 2-64424509543 3-519691042818 4-107374182487 5-523986010114 6-150323855432 7-171798691906 8-193273528377' 2024-02-05T05:43:35.649 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:35.649 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:43:35.650 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631503 2024-02-05T05:43:35.650 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631503 2024-02-05T05:43:35.650 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673069 2-64424509543 3-519691042818 4-107374182487 5-523986010114 6-150323855432 7-171798691906 8-193273528377 9-442381631503' 2024-02-05T05:43:35.650 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:35.650 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836598 2024-02-05T05:43:35.650 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:35.650 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:43:35.650 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836598 2024-02-05T05:43:35.650 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:35.650 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836598 2024-02-05T05:43:35.650 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836598' 2024-02-05T05:43:35.650 INFO:tasks.workunit.client.0.smithi148.stderr:231286: waiting osd.0 seq 21474836598 2024-02-05T05:43:35.650 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:35.650 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836599 -lt 21474836598 2024-02-05T05:43:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673069 2024-02-05T05:43:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:43:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673069 2024-02-05T05:43:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673069 2024-02-05T05:43:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673069' 2024-02-05T05:43:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:231286: waiting osd.1 seq 42949673069 2024-02-05T05:43:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:43:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673071 -lt 42949673069 2024-02-05T05:43:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509543 2024-02-05T05:43:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:35.910 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/ceptu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528378 2024-02-05T05:43:35.910 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528378 2024-02-05T05:43:35.910 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673071 2-64424509544 3-519691042819 4-107374182488 5-523986010115 6-150323855433 7-171798691907 8-193273528378' 2024-02-05T05:43:35.910 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:35.910 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:43:35.910 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631504 2024-02-05T05:43:35.910 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631504 2024-02-05T05:43:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673071 2-64424509544 3-519691042819 4-107374182488 5-523986010115 6-150323855433 7-171798691907 8-193273528378 9-442381631504' 2024-02-05T05:43:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836599 2024-02-05T05:43:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:43:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836599 2024-02-05T05:43:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836599 2024-02-05T05:43:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836599' 2024-02-05T05:43:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:231288: waiting osd.0 seq 21474836599 2024-02-05T05:43:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836599 -lt 21474836599 2024-02-05T05:43:35.911 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673071 2024-02-05T05:43:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:43:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673071 2024-02-05T05:43:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673071 2024-02-05T05:43:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673071' 2024-02-05T05:43:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:231288: waiting osd.1 seq 42949673071 2024-02-05T05:43:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:43:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673071 -lt 42949673071 2024-02-05T05:43:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509544 2024-02-05T05:43:35.912 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:43:36.592 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntuhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:43:36.592 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509543 2024-02-05T05:43:36.592 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:36.592 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509543 2024-02-05T05:43:36.592 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509543' 2024-02-05T05:43:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:231286: waiting osd.2 seq 64424509543 2024-02-05T05:43:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:43:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509544 -lt 64424509543 2024-02-05T05:43:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-519691042818 2024-02-05T05:43:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:43:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-519691042818 2024-02-05T05:43:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=519691042818 2024-02-05T05:43:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 519691042818' 2024-02-05T05:43:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:231286: waiting osd.3 seq 519691042818 2024-02-05T05:43:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:43:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 519691042820 -lt 519691042818 2024-02-05T05:43:36.594 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:36.594 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182487 2024-02-05T05:43:36.594 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:36.594 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:43:36.594 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182487 2024-02-05T05:43:36.594 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:36.594 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182487 2024-02-05T05:43:36.594 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182487' 2024-02-05T05:43:36.594 INFO:tasks.workunit.client.0.smithi148.stderr:231286: waiting osd.4 seq 107374182487 2024-02-05T05:43:36.594 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:43:36.594 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182488 -lt 107374182487 2024-02-05T05:43:36.594 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:36.595 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-523986010114 2024-02-05T05:43:36.595 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:36.595 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:43:36.595 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-523986010114 2024-02-05T05:43:36.595 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:36.881 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=5239/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509544 2024-02-05T05:43:36.881 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:36.881 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509544 2024-02-05T05:43:36.881 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509544' 2024-02-05T05:43:36.881 INFO:tasks.workunit.client.0.smithi148.stderr:231288: waiting osd.2 seq 64424509544 2024-02-05T05:43:36.881 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:43:36.881 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509544 -lt 64424509544 2024-02-05T05:43:36.881 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:36.881 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-519691042819 2024-02-05T05:43:36.881 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:43:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-519691042819 2024-02-05T05:43:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=519691042819 2024-02-05T05:43:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 519691042819' 2024-02-05T05:43:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:231288: waiting osd.3 seq 519691042819 2024-02-05T05:43:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:43:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 519691042820 -lt 519691042819 2024-02-05T05:43:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182488 2024-02-05T05:43:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:43:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182488 2024-02-05T05:43:36.883 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:36.883 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182488 2024-02-05T05:43:36.883 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182488' 2024-02-05T05:43:36.883 INFO:tasks.workunit.client.0.smithi148.stderr:231288: waiting osd.4 seq 107374182488 2024-02-05T05:43:36.883 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:43:36.883 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182488 -lt 107374182488 2024-02-05T05:43:36.883 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:36.883 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-523986010115 2024-02-05T05:43:36.883 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:36.883 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:43:36.883 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-523986010115 2024-02-05T05:43:36.883 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:36.883 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=523986010115 2024-02-05T05:43:37.928 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_st86010114 2024-02-05T05:43:37.928 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 523986010114' 2024-02-05T05:43:37.928 INFO:tasks.workunit.client.0.smithi148.stderr:231286: waiting osd.5 seq 523986010114 2024-02-05T05:43:37.928 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:43:37.929 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010115 -lt 523986010114 2024-02-05T05:43:37.929 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:37.929 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855432 2024-02-05T05:43:37.929 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:37.929 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:43:37.929 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855432 2024-02-05T05:43:37.929 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:37.929 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855432 2024-02-05T05:43:37.929 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855432' 2024-02-05T05:43:37.929 INFO:tasks.workunit.client.0.smithi148.stderr:231286: waiting osd.6 seq 150323855432 2024-02-05T05:43:37.929 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:43:37.929 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855433 -lt 150323855432 2024-02-05T05:43:37.929 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:37.929 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:37.930 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691906 2024-02-05T05:43:37.930 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:43:37.930 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691906 2024-02-05T05:43:37.930 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:37.930 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691906 2024-02-05T05:43:37.930 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691906' 2024-02-05T05:43:37.930 INFO:tasks.workunit.client.0.smithi148.stderr:231286: waiting osd.7 seq 171798691906 2024-02-05T05:43:37.930 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:43:37.930 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691908 -lt 171798691906 2024-02-05T05:43:37.930 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:37.930 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:37.930 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528377 2024-02-05T05:43:37.931 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:43:37.931 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528377 2024-02-05T05:43:37.931 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:37.931 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528377 2024-02-05T05:43:37.931 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528377' 2024-02-05T05:43:37.931 INFO:tasks.workunit.client.0.smithi148.stderr:231286: waiting osd.8 seq 193273528377 2024-02-05T05:43:37.931 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:43:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1ats: echo 'waiting osd.5 seq 523986010115' 2024-02-05T05:43:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:231288: waiting osd.5 seq 523986010115 2024-02-05T05:43:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:43:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010115 -lt 523986010115 2024-02-05T05:43:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855433 2024-02-05T05:43:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:43:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855433 2024-02-05T05:43:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855433 2024-02-05T05:43:38.200 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855433' 2024-02-05T05:43:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:231288: waiting osd.6 seq 150323855433 2024-02-05T05:43:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:43:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855433 -lt 150323855433 2024-02-05T05:43:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691907 2024-02-05T05:43:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:43:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691907 2024-02-05T05:43:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691907 2024-02-05T05:43:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691907' 2024-02-05T05:43:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:231288: waiting osd.7 seq 171798691907 2024-02-05T05:43:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:43:38.201 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691908 -lt 171798691907 2024-02-05T05:43:38.202 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:38.202 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:38.202 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528378 2024-02-05T05:43:38.202 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:43:38.202 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528378 2024-02-05T05:43:38.202 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:38.202 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528378 2024-02-05T05:43:38.202 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528378' 2024-02-05T05:43:38.202 INFO:tasks.workunit.client.0.smithi148.stderr:231288: waiting osd.8 seq 193273528378 2024-02-05T05:43:38.202 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:43:38.202 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528378 -lt 193273528378 2024-02-05T05:43:39.373 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.93273528378 -lt 193273528377 2024-02-05T05:43:39.374 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:39.374 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631503 2024-02-05T05:43:39.374 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:39.374 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:43:39.374 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631503 2024-02-05T05:43:39.374 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:39.374 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631503 2024-02-05T05:43:39.374 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631503' 2024-02-05T05:43:39.374 INFO:tasks.workunit.client.0.smithi148.stderr:231286: waiting osd.9 seq 442381631503 2024-02-05T05:43:39.374 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:43:39.374 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631504 -lt 442381631503 2024-02-05T05:43:39.375 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:43:39.375 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:39.375 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:39.375 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:43:39.375 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:39.375 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:39.375 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:39.375 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:39.375 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:39.375 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:39.375 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:39.375 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:43:39.375 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:39.375 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:39.376 INFO:tasks.workunit.client.0.smithi148.stderr:231286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:39.376 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:43:39.376 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:43:39.376 INFO:tasks.workunit.client.0.smithi148.stderr:231286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:43:39.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:43:39.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:43:39.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 231285 2024-02-05T05:43:39.692 INFO:tasks.workunit.client.0.smithi148.stderr:sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:39.692 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631504 2024-02-05T05:43:39.693 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:39.693 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:43:39.693 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631504 2024-02-05T05:43:39.693 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:39.693 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631504 2024-02-05T05:43:39.693 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631504' 2024-02-05T05:43:39.693 INFO:tasks.workunit.client.0.smithi148.stderr:231288: waiting osd.9 seq 442381631504 2024-02-05T05:43:39.693 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:43:39.693 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631504 -lt 442381631504 2024-02-05T05:43:39.693 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:43:39.693 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:39.693 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:39.693 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:43:39.694 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:39.694 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:39.694 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:39.694 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:39.694 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:39.694 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:39.694 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:39.694 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:43:39.694 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:39.694 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:39.694 INFO:tasks.workunit.client.0.smithi148.stderr:231288: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:39.694 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:43:39.694 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:43:39.695 INFO:tasks.workunit.client.0.smithi148.stderr:231288: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:43:39.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:43:39.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:43:39.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:43:39.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:43:39.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:43:39.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:43:39.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:43:39.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:43:39.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:43:39.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:43:39.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:43:39.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:43:39.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:43:39.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:43:39.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:43:39.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:43:39.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:43:40.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:43:40.087 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:43:40.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:43:40.089 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:43:40.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:43:40.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:43:40.091 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:43:40.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:43:40.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:43:40.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:43:40.093 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:43:40.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:43:40.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:43:40.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:43:40.174 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:43:40.174 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:40.174 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:43:40.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:43:40.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:43:40.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:43:40.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:43:40.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:43:40.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:43:40.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:43:40.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:43:40.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:43:40.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:43:40.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:43:40.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:43:40.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:43:40.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:43:40.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:43:40.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:43:40.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:43:40.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:43:40.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:43:40.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:43:40.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:43:40.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:43:40.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:43:40.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:43:40.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:43:40.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:43:40.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:43:40.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:43:40.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:43:40.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:43:40.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:40.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:43:40.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:43:40.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:43:40.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:43:40.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:43:40.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:43:40.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:40.193 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:43:40.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T05:43:40.195 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:43:40.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:43:40.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:43:40.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:43:40.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-02-05T05:43:40.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-02-05T05:43:40.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-02-05T05:43:40.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-02-05T05:43:40.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-02-05T05:43:40.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-02-05T05:43:40.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-02-05T05:43:40.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:43:40.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:43:40.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:43:40.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:43:40.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:43:40.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:43:40.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:43:40.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:43:40.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:43:40.610 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:43:40.610 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:40.610 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:43:40.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:43:40.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:43:40.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:43:40.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:43:40.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:43:40.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:43:40.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:43:40.821 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:43:40.821 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:43:40.821 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:43:40.821 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:43:40.822 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:43:40.822 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:40.822 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:43:40.822 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:43:40.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:43:40.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T05:43:40.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:43:40.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:43:40.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:43:40.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:43:40.937 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:43:40.938 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:43:40.938 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:43:40.938 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:43:40.938 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:43:40.938 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:40.938 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:43:40.938 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:43:40.938 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:43:40.938 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T05:43:41.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-02-05T05:43:41.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:43:41.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:43:41.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:43:41.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:43:41.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:43:41.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:43:41.897 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:43:41.897 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:43:41.897 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:43:41.897 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:43:41.897 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:41.897 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:43:41.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:43:41.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:43:41.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:43:41.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-02-05T05:43:41.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:43:41.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-02-05T05:43:41.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:43:41.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:43:41.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:43:41.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:43:41.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:43:41.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:43:41.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:43:41.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:43:41.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:43:41.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:43:41.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:43:41.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:43:41.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:43:41.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:43:41.938 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:43:41.938 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:43:41.938 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:43:41.938 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:43:41.938 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:41.938 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:43:41.938 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:43:41.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:43:41.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:43:41.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:43:41.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:43:41.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:43:41.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:43:41.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:43:41.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:43:41.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:43:41.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:43:41.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:43:41.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:43:41.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:43:41.940 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:43:41.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8c79a825-0c2b-401a-9e78-af587af5113f 2024-02-05T05:43:41.940 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 8c79a825-0c2b-401a-9e78-af587af5113f 2024-02-05T05:43:41.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8c79a825-0c2b-401a-9e78-af587af5113f' 2024-02-05T05:43:41.941 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:43:41.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCNdcBl1s9NORAAJO/Nfbwik75BojUOARgqAg== 2024-02-05T05:43:41.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCNdcBl1s9NORAAJO/Nfbwik75BojUOARgqAg=="}' 2024-02-05T05:43:41.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8c79a825-0c2b-401a-9e78-af587af5113f -i td/osd-scrub-repair/0/new.json 2024-02-05T05:43:42.224 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:43:42.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:43:42.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCNdcBl1s9NORAAJO/Nfbwik75BojUOARgqAg== --osd-uuid 8c79a825-0c2b-401a-9e78-af587af5113f 2024-02-05T05:43:42.271 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:43:42.273+0000 7feaa9c47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:42.292 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:43:42.293+0000 7feaa9c47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:42.293 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:43:42.297+0000 7feaa9c47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:42.293 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:43:42.297+0000 7feaa9c47640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:43:42.293 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:43:42.297+0000 7feaa9c47640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:43:42.293 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:43:42.297+0000 7feaa9c47640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:43:42.384 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:43:42.385+0000 7feaa9c47640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:43:48.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:43:48.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:43:48.639 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:43:48.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:43:48.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:43:49.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:43:49.015 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:43:49.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:43:49.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:43:49.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:43:49.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:43:49.046 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:43:49.049+0000 7f0ace1ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:49.048 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:43:49.053+0000 7f0ace1ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:49.050 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:43:49.053+0000 7f0ace1ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:49.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:43:49.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:43:49.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:43:49.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:43:49.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:43:49.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:49.350 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:43:49.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:43:49.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:49.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:49.599 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:43:49.601+0000 7f0ace1ca640 -1 Falling back to public interface 2024-02-05T05:43:49.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:50.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:50.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:50.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:43:50.679 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:43:50.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:50.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:50.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:51.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:51.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:51.985 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:43:51.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:43:51.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:51.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:52.180 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:43:52.185+0000 7f0ace1ca640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:43:52.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:53.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:53.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:53.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:43:53.296 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:43:53.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:53.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:53.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:54.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:54.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:54.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:43:54.618 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:43:54.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:54.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:55.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:56.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:56.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:56.017 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:43:56.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:43:56.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:56.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:56.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:57.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:57.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:57.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:43:57.357 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:43:57.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:57.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:57.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:58.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:58.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:58.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:43:58.669 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:43:58.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:58.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:59.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:00.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:00.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:00.036 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:44:00.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:44:00.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:00.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:00.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:01.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:01.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:01.353 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:44:01.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:44:01.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:01.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:01.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:02.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:02.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:02.678 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:44:02.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:44:02.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:02.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:03.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:04.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:04.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:04.032 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:44:04.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:44:04.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:04.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:04.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:05.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:05.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:05.350 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:44:05.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:44:05.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:05.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:05.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:06.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:06.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:06.657 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:44:06.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:44:06.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:06.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:07.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:08.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:08.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:08.119 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:44:08.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:44:08.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:08.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:08.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:09.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:09.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:09.910 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:44:09.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:44:09.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:09.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:10.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:11.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:11.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:11.637 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:44:11.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:44:11.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:11.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:11.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:12.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:12.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:12.959 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:44:12.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:44:12.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:12.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:13.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:14.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:14.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:14.474 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:44:14.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:44:14.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:14.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:15.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:16.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:16.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:16.036 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:44:16.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:44:16.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:16.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:16.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:17.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:17.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:17.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:44:17.745 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:44:17.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:17.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:18.051 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2823096144,v1:127.0.0.1:6803/2823096144] [v2:127.0.0.1:6804/2823096144,v1:127.0.0.1:6805/2823096144] exists,up 8c79a825-0c2b-401a-9e78-af587af5113f 2024-02-05T05:44:18.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:44:18.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:44:18.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:44:18.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:44:18.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-02-05T05:44:18.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:44:18.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:44:18.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:44:18.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:44:18.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:44:18.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:44:18.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:44:18.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:44:18.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:44:18.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:44:18.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:44:18.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:44:18.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:44:18.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:44:18.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:44:18.062 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:44:18.062 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:44:18.062 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:44:18.062 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:18.063 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:44:18.063 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:44:18.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:44:18.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:44:18.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:44:18.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:44:18.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:44:18.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:44:18.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:44:18.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:44:18.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:44:18.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:44:18.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:44:18.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:44:18.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:44:18.065 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:44:18.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4c2423ac-a1c7-407d-b41d-7f18b422f141 2024-02-05T05:44:18.066 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 4c2423ac-a1c7-407d-b41d-7f18b422f141 2024-02-05T05:44:18.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4c2423ac-a1c7-407d-b41d-7f18b422f141' 2024-02-05T05:44:18.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:44:18.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCydcBl8ZzVBBAAACOv2YDNQMHwF+m1N8p8lg== 2024-02-05T05:44:18.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCydcBl8ZzVBBAAACOv2YDNQMHwF+m1N8p8lg=="}' 2024-02-05T05:44:18.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4c2423ac-a1c7-407d-b41d-7f18b422f141 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:44:18.547 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:44:18.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:44:18.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCydcBl8ZzVBBAAACOv2YDNQMHwF+m1N8p8lg== --osd-uuid 4c2423ac-a1c7-407d-b41d-7f18b422f141 2024-02-05T05:44:18.604 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:18.608+0000 7f38d4ea3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:18.648 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:18.652+0000 7f38d4ea3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:18.650 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:18.652+0000 7f38d4ea3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:18.651 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:18.652+0000 7f38d4ea3640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:44:18.651 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:18.652+0000 7f38d4ea3640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:44:18.651 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:18.652+0000 7f38d4ea3640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:44:18.732 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:18.736+0000 7f38d4ea3640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:44:25.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:44:25.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:44:25.395 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:44:25.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:44:25.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:44:25.768 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:44:25.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:44:25.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:44:25.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:44:25.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:44:25.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:44:25.801 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:25.804+0000 7f389f43c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:25.837 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:25.840+0000 7f389f43c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:25.839 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:25.840+0000 7f389f43c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:26.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:44:26.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:44:26.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:44:26.112 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:44:26.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:44:26.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:44:26.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:26.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:44:26.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:26.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:26.397 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:26.400+0000 7f389f43c640 -1 Falling back to public interface 2024-02-05T05:44:26.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:27.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:27.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:27.441 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:44:27.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:44:27.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:27.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:27.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:28.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:28.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:28.751 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:44:28.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:44:28.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:28.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:29.022 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:29.024+0000 7f389f43c640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:44:29.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:30.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:30.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:30.067 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:44:30.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:44:30.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:30.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:30.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:31.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:31.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:31.470 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:44:31.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:44:31.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:31.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:31.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:32.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:32.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:32.801 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:44:32.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:44:32.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:32.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:33.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:34.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:34.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:34.352 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:44:34.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:44:34.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:34.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:34.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:35.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:35.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:35.671 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:44:35.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:44:35.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:35.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:36.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:37.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:37.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:37.059 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:44:37.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:44:37.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:37.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:37.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:38.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:38.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:38.367 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:44:38.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:44:38.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:38.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:38.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:39.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:39.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:39.667 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:44:39.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:44:39.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:39.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:40.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:41.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:41.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:41.020 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:44:41.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:44:41.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:41.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:41.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:42.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:42.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:42.341 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:44:42.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:44:42.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:42.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:42.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:43.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:43.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:43.661 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:44:43.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:44:43.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:43.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:44.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:45.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:45.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:45.450 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:44:45.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:44:45.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:45.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:45.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:46.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:46.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:46.862 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:44:46.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:44:46.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:46.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:47.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:48.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:48.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:48.287 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:44:48.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:44:48.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:48.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:48.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:49.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:49.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:49.812 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:44:49.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:44:49.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:49.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:50.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:51.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:51.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:51.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:44:51.390 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:44:51.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:51.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:52.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:53.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:53.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:53.033 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:44:53.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:44:53.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:53.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:53.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:54.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:54.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:54.557 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:44:54.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:44:54.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:54.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:54.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:55.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:55.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:55.928 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T05:44:55.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T05:44:55.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:55.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:56.283 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3054458388,v1:127.0.0.1:6811/3054458388] [v2:127.0.0.1:6812/3054458388,v1:127.0.0.1:6813/3054458388] exists,up 4c2423ac-a1c7-407d-b41d-7f18b422f141 2024-02-05T05:44:56.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:44:56.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:44:56.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:44:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:44:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-02-05T05:44:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:44:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:44:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:44:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:44:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:44:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:44:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:44:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:44:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:44:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:44:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:44:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:44:56.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:44:56.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:44:56.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:44:56.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:44:56.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:44:56.287 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:44:56.287 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:56.287 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:44:56.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:44:56.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:44:56.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:44:56.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:44:56.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:44:56.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:44:56.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:44:56.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:44:56.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:44:56.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:44:56.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:44:56.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:44:56.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:44:56.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:44:56.290 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:44:56.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=65ae0981-1fa8-4caa-affa-6c3641971f06 2024-02-05T05:44:56.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 65ae0981-1fa8-4caa-affa-6c3641971f06' 2024-02-05T05:44:56.291 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 65ae0981-1fa8-4caa-affa-6c3641971f06 2024-02-05T05:44:56.291 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:44:56.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDYdcBlRtvGEhAAhvG6+IJJC5uNGZQRbBNDHw== 2024-02-05T05:44:56.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDYdcBlRtvGEhAAhvG6+IJJC5uNGZQRbBNDHw=="}' 2024-02-05T05:44:56.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 65ae0981-1fa8-4caa-affa-6c3641971f06 -i td/osd-scrub-repair/2/new.json 2024-02-05T05:44:56.730 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:44:56.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:44:56.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYdcBlRtvGEhAAhvG6+IJJC5uNGZQRbBNDHw== --osd-uuid 65ae0981-1fa8-4caa-affa-6c3641971f06 2024-02-05T05:44:56.785 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:56.787+0000 7f3038768640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:56.833 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:56.835+0000 7f3038768640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:56.834 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:56.839+0000 7f3038768640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:56.835 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:56.839+0000 7f3038768640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:44:56.835 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:56.839+0000 7f3038768640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:44:56.835 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:56.839+0000 7f3038768640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:44:56.916 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:44:56.919+0000 7f3038768640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:45:03.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:45:03.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:45:03.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:45:03.774 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T05:45:03.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:45:04.159 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T05:45:04.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:45:04.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:45:04.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:45:04.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:45:04.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:45:04.192 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:04.195+0000 7fa99af30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:04.222 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:04.227+0000 7fa99af30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:04.226 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:04.227+0000 7fa99af30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:04.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:45:04.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:45:04.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:45:04.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:45:04.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:45:04.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:04.488 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:45:04.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:45:04.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:04.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:04.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:04.783 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:04.787+0000 7fa99af30640 -1 Falling back to public interface 2024-02-05T05:45:05.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:05.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:05.784 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:45:05.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:45:05.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:05.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:06.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:07.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:07.092 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:45:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:45:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:07.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:07.677 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:07.679+0000 7fa99af30640 -1 osd.2 0 log_to_monitors true 2024-02-05T05:45:08.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:08.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:08.502 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:45:08.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:45:08.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:08.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:08.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:09.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:09.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:09.810 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:45:09.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:45:09.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:09.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:10.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:11.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:11.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:11.113 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:45:11.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:45:11.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:11.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:11.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:12.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:12.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:12.487 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:45:12.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:45:12.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:12.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:12.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:13.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:13.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:13.813 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:45:13.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:45:13.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:13.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:14.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:15.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:15.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:15.132 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:45:15.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:45:15.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:15.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:15.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:16.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:16.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:16.478 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:45:16.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:45:16.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:16.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:16.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:17.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:17.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:17.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:45:17.848 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:45:17.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:17.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:18.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:19.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:19.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:19.201 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:45:19.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:45:19.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:19.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:19.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:20.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:20.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:20.507 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:45:20.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:45:20.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:20.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:20.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:21.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:21.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:21.825 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:45:21.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:45:21.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:21.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:22.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:23.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:23.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:23.211 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:45:23.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:45:23.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:23.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:23.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:24.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:24.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:24.532 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:45:24.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:45:24.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:24.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:25.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:26.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:26.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:26.352 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:45:26.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:45:26.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:26.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:26.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:27.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:27.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:27.889 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:45:27.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:45:27.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:27.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:28.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:29.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:29.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:29.766 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:45:29.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:45:29.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:29.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:30.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:31.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:31.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:31.158 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:45:31.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:45:31.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:31.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:31.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:32.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:32.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:32.986 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:45:32.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:45:32.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:32.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:33.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:34.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:34.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:34.408 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T05:45:34.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T05:45:34.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:34.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:45:34.710 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2405040193,v1:127.0.0.1:6819/2405040193] [v2:127.0.0.1:6820/2405040193,v1:127.0.0.1:6821/2405040193] exists,up 65ae0981-1fa8-4caa-affa-6c3641971f06 2024-02-05T05:45:34.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:45:34.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:45:34.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:45:34.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:45:34.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-02-05T05:45:34.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:45:34.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:45:34.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T05:45:34.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:45:34.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:45:34.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:45:34.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:45:34.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:45:34.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:45:34.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:45:34.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:45:34.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:45:34.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:45:34.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:45:34.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:45:34.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:45:34.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:45:34.714 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:45:34.714 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:45:34.714 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:45:34.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:45:34.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:45:34.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:45:34.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:45:34.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:45:34.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:45:34.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:45:34.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:45:34.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:45:34.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:45:34.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:45:34.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:45:34.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:45:34.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:45:34.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:45:34.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5498ffc6-6c90-4948-b650-4948ddc04afa 2024-02-05T05:45:34.718 INFO:tasks.workunit.client.0.smithi148.stdout:add osd3 5498ffc6-6c90-4948-b650-4948ddc04afa 2024-02-05T05:45:34.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5498ffc6-6c90-4948-b650-4948ddc04afa' 2024-02-05T05:45:34.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:45:34.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+dcBlq/QmLBAAMKuEBE7GyHu35y6oNmwfSA== 2024-02-05T05:45:34.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+dcBlq/QmLBAAMKuEBE7GyHu35y6oNmwfSA=="}' 2024-02-05T05:45:34.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5498ffc6-6c90-4948-b650-4948ddc04afa -i td/osd-scrub-repair/3/new.json 2024-02-05T05:45:35.193 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:45:35.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-02-05T05:45:35.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+dcBlq/QmLBAAMKuEBE7GyHu35y6oNmwfSA== --osd-uuid 5498ffc6-6c90-4948-b650-4948ddc04afa 2024-02-05T05:45:35.242 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:35.246+0000 7f3948a8f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:35.305 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:35.306+0000 7f3948a8f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:35.306 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:35.310+0000 7f3948a8f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:35.307 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:35.310+0000 7f3948a8f640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:45:35.307 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:35.310+0000 7f3948a8f640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:45:35.307 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:35.310+0000 7f3948a8f640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:45:35.386 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:35.390+0000 7f3948a8f640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-02-05T05:45:42.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-02-05T05:45:42.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:45:42.356 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd3 key to auth repository 2024-02-05T05:45:42.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T05:45:42.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:45:42.747 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.3 2024-02-05T05:45:42.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T05:45:42.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:45:42.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:45:42.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:45:42.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:45:42.780 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:42.782+0000 7f438122f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:42.818 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:42.822+0000 7f438122f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:42.819 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:42.822+0000 7f438122f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:43.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T05:45:43.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:45:43.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:45:43.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:45:43.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:45:43.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:43.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:45:43.140 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:45:43.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:43.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:45:43.395 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:43.398+0000 7f438122f640 -1 Falling back to public interface 2024-02-05T05:45:43.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:44.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:44.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:44.515 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:45:44.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:45:44.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:44.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:45:44.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:45.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:45.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:45.829 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:45:45.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:45:45.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:45.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:45:46.007 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:45:46.010+0000 7f438122f640 -1 osd.3 0 log_to_monitors true 2024-02-05T05:45:46.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:47.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:47.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:47.143 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:45:47.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:45:47.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:47.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:45:47.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:48.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:48.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:48.492 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:45:48.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:45:48.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:48.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:45:48.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:49.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:49.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:49.817 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:45:49.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:45:49.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:49.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:45:50.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:51.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:51.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:51.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:45:51.137 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:45:51.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:51.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:45:51.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:52.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:52.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:52.590 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:45:52.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:45:52.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:52.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:45:52.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:53.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:53.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:53.988 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:45:53.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:45:53.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:53.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:45:54.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:55.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:55.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:55.293 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:45:55.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:45:55.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:55.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:45:55.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:56.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:56.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:56.705 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:45:56.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:45:56.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:56.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:45:57.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:58.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:58.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:58.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:45:58.094 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:45:58.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:58.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:45:58.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:59.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:59.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:59.484 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:45:59.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:45:59.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:59.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:45:59.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:00.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:00.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:00.959 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:46:00.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:46:00.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:00.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:46:01.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:02.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:02.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:02.476 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:46:02.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:46:02.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:02.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:46:03.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:04.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:04.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:04.433 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:46:04.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:46:04.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:04.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:46:04.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:05.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:05.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:05.966 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:46:05.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:46:05.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:05.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:46:06.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:07.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:07.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:07.753 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:46:07.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:46:07.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:07.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:46:08.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:09.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:09.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:09.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:46:09.355 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:46:09.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:09.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:46:10.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:11.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:11.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:11.226 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:46:11.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:46:11.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:11.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:46:11.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:12.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:12.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:12.628 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:46:12.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:46:12.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:12.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:46:13.119 INFO:tasks.workunit.client.0.smithi148.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1471508413,v1:127.0.0.1:6827/1471508413] [v2:127.0.0.1:6828/1471508413,v1:127.0.0.1:6829/1471508413] exists,up 5498ffc6-6c90-4948-b650-4948ddc04afa 2024-02-05T05:46:13.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:46:13.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:46:13.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:46:13.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:46:13.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-02-05T05:46:13.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:46:13.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:46:13.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-02-05T05:46:13.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:46:13.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-02-05T05:46:13.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:46:13.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:46:13.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:46:13.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:46:13.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-02-05T05:46:13.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-02-05T05:46:13.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:46:13.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:46:13.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:46:13.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:46:13.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:46:13.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:46:13.124 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:46:13.124 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:46:13.124 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:46:13.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:46:13.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:46:13.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:46:13.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:46:13.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:46:13.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:46:13.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:46:13.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:46:13.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:46:13.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:46:13.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:46:13.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:46:13.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:46:13.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-02-05T05:46:13.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:46:13.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d0694934-fb04-41ca-8daf-00f5728c90dc 2024-02-05T05:46:13.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 d0694934-fb04-41ca-8daf-00f5728c90dc' 2024-02-05T05:46:13.126 INFO:tasks.workunit.client.0.smithi148.stdout:add osd4 d0694934-fb04-41ca-8daf-00f5728c90dc 2024-02-05T05:46:13.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:46:13.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAldsBlbkVVCRAA9ROUB8zFlWzPhu9K4GrCWw== 2024-02-05T05:46:13.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAldsBlbkVVCRAA9ROUB8zFlWzPhu9K4GrCWw=="}' 2024-02-05T05:46:13.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d0694934-fb04-41ca-8daf-00f5728c90dc -i td/osd-scrub-repair/4/new.json 2024-02-05T05:46:13.522 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:46:13.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-02-05T05:46:13.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAldsBlbkVVCRAA9ROUB8zFlWzPhu9K4GrCWw== --osd-uuid d0694934-fb04-41ca-8daf-00f5728c90dc 2024-02-05T05:46:13.578 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:13.585+0000 7f19ff78e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:13.633 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:13.641+0000 7f19ff78e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:13.634 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:13.641+0000 7f19ff78e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:13.635 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:13.641+0000 7f19ff78e640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-02-05T05:46:13.635 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:13.641+0000 7f19ff78e640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-02-05T05:46:13.635 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:13.641+0000 7f19ff78e640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-02-05T05:46:13.712 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:13.721+0000 7f19ff78e640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-02-05T05:46:20.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-02-05T05:46:20.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:46:20.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-02-05T05:46:20.691 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd4 key to auth repository 2024-02-05T05:46:20.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:46:21.131 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.4 2024-02-05T05:46:21.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-02-05T05:46:21.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:46:21.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:46:21.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:46:21.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:46:21.164 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:21.173+0000 7f419e826640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:21.184 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:21.193+0000 7f419e826640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:21.185 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:21.193+0000 7f419e826640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:21.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-02-05T05:46:21.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:46:21.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-02-05T05:46:21.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:46:21.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:46:21.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:21.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:46:21.444 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:46:21.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:21.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:21.734 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:21.741+0000 7f419e826640 -1 Falling back to public interface 2024-02-05T05:46:21.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:22.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:22.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:22.757 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:46:22.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:46:22.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:22.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:23.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:24.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:24.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:24.070 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:46:24.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:46:24.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:24.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:24.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:24.400 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:24.409+0000 7f419e826640 -1 osd.4 0 log_to_monitors true 2024-02-05T05:46:25.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:25.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:25.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:46:25.382 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:46:25.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:25.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:25.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:26.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:26.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:26.811 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:46:26.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:46:26.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:26.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:27.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:28.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:28.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:28.124 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:46:28.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:46:28.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:28.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:28.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:29.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:29.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:29.584 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:46:29.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:46:29.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:29.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:29.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:30.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:30.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:30.961 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:46:30.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:46:30.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:30.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:31.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:32.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:32.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:32.313 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:46:32.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:46:32.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:32.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:32.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:33.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:33.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:33.630 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:46:33.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:46:33.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:33.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:34.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:35.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:35.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:35.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:46:35.020 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:46:35.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:35.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:35.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:36.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:36.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:36.359 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:46:36.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:46:36.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:36.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:36.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:37.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:37.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:37.706 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:46:37.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:46:37.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:37.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:38.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:39.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:39.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:39.026 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:46:39.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:46:39.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:39.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:40.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:41.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:41.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:41.169 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:46:41.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:46:41.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:41.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:41.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:42.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:42.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:42.925 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:46:42.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:46:42.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:42.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:44.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:45.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:45.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:45.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:46:45.045 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:46:45.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:45.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:45.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:46.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:46.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:46.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:46:46.664 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:46:46.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:46.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:47.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:48.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:48.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:48.351 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:46:48.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:46:48.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:48.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:49.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:50.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:50.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:50.003 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:46:50.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:46:50.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:50.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:50.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:51.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:51.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:51.354 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:46:51.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:46:51.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:51.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:46:51.669 INFO:tasks.workunit.client.0.smithi148.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/610557074,v1:127.0.0.1:6835/610557074] [v2:127.0.0.1:6836/610557074,v1:127.0.0.1:6837/610557074] exists,up d0694934-fb04-41ca-8daf-00f5728c90dc 2024-02-05T05:46:51.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:46:51.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:46:51.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:46:51.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:46:51.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-02-05T05:46:51.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:46:51.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:46:51.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-02-05T05:46:51.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:46:51.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:46:51.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:46:51.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:46:51.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:46:51.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:46:51.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:46:51.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:46:51.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:46:51.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:46:51.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:46:51.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:46:51.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:46:51.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:46:51.677 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:46:51.677 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:46:51.677 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:46:51.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:46:51.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:46:51.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:46:51.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:46:51.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:46:51.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:46:51.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:46:51.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:46:51.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:46:51.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:46:51.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:46:51.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:46:51.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:46:51.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:46:51.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:46:51.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9832157c-d831-4b53-9b05-71ea506d914b 2024-02-05T05:46:51.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 9832157c-d831-4b53-9b05-71ea506d914b' 2024-02-05T05:46:51.679 INFO:tasks.workunit.client.0.smithi148.stdout:add osd5 9832157c-d831-4b53-9b05-71ea506d914b 2024-02-05T05:46:51.679 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:46:51.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBLdsBl9Y4pKhAAkt07v+yqy2NiOE3XQYYvaw== 2024-02-05T05:46:51.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBLdsBl9Y4pKhAAkt07v+yqy2NiOE3XQYYvaw=="}' 2024-02-05T05:46:51.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9832157c-d831-4b53-9b05-71ea506d914b -i td/osd-scrub-repair/5/new.json 2024-02-05T05:46:52.045 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:46:52.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-02-05T05:46:52.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLdsBl9Y4pKhAAkt07v+yqy2NiOE3XQYYvaw== --osd-uuid 9832157c-d831-4b53-9b05-71ea506d914b 2024-02-05T05:46:52.097 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:52.105+0000 7f3f508b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:52.121 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:52.129+0000 7f3f508b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:52.122 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:52.129+0000 7f3f508b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:52.122 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:52.129+0000 7f3f508b4640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-02-05T05:46:52.123 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:52.129+0000 7f3f508b4640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-02-05T05:46:52.123 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:52.129+0000 7f3f508b4640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-02-05T05:46:52.237 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:52.245+0000 7f3f508b4640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-02-05T05:46:59.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-02-05T05:46:59.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:46:59.200 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd5 key to auth repository 2024-02-05T05:46:59.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-02-05T05:46:59.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:46:59.575 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.5 2024-02-05T05:46:59.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-02-05T05:46:59.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:46:59.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:46:59.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:46:59.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:46:59.608 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:59.616+0000 7f5253f0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:59.639 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:59.648+0000 7f5253f0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:59.640 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:46:59.648+0000 7f5253f0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:59.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-02-05T05:46:59.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:46:59.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:46:59.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:46:59.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:46:59.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:59.915 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:46:59.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:46:59.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:59.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:00.209 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:47:00.216+0000 7f5253f0f640 -1 Falling back to public interface 2024-02-05T05:47:00.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:01.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:01.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:01.255 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:47:01.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:47:01.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:01.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:01.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:02.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:02.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:02.566 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:47:02.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:47:02.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:02.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:02.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:02.883 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:47:02.892+0000 7f5253f0f640 -1 osd.5 0 log_to_monitors true 2024-02-05T05:47:03.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:03.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:03.868 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:47:03.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:47:03.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:03.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:04.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:05.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:05.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:05.312 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:47:05.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:47:05.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:05.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:05.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:06.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:06.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:06.772 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:47:06.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:47:06.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:06.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:07.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:08.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:08.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:08.126 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:47:08.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:47:08.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:08.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:08.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:09.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:09.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:09.498 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:47:09.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:47:09.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:09.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:10.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:11.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:11.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:11.069 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:47:11.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:47:11.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:11.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:11.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:12.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:12.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:12.500 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:47:12.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:47:12.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:12.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:12.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:13.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:13.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:13.963 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:47:13.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:47:13.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:13.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:14.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:15.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:15.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:15.281 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:47:15.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:47:15.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:15.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:15.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:16.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:16.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:16.665 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:47:16.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:47:16.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:16.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:17.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:18.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:18.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:18.156 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:47:18.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:47:18.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:18.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:18.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:19.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:19.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:19.588 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:47:19.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:47:19.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:19.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:20.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:21.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:21.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:21.243 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:47:21.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:47:21.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:21.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:21.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:22.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:22.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:22.688 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:47:22.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:47:22.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:22.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:23.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:24.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:24.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:24.372 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:47:24.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:47:24.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:24.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:24.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:25.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:25.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:25.895 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:47:25.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:47:25.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:25.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:26.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:27.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:27.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:27.360 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:47:27.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:47:27.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:27.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:27.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:28.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:28.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:28.851 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:47:28.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:47:28.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:28.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:29.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:30.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:30.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:30.628 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T05:47:30.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T05:47:30.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:30.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:47:30.947 INFO:tasks.workunit.client.0.smithi148.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1161205995,v1:127.0.0.1:6843/1161205995] [v2:127.0.0.1:6844/1161205995,v1:127.0.0.1:6845/1161205995] exists,up 9832157c-d831-4b53-9b05-71ea506d914b 2024-02-05T05:47:30.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:47:30.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:47:30.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:47:30.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:47:30.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-02-05T05:47:30.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:47:30.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:47:30.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-02-05T05:47:30.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:47:30.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-02-05T05:47:30.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:47:30.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:47:30.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:47:30.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:47:30.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-02-05T05:47:30.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-02-05T05:47:30.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:47:30.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:47:30.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:47:30.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:47:30.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:47:30.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:47:30.953 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:47:30.953 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:47:30.953 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:47:30.953 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:47:30.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:47:30.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:47:30.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:47:30.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:47:30.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:47:30.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:47:30.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:47:30.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:47:30.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:47:30.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:47:30.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:47:30.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:47:30.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-02-05T05:47:30.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:47:30.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c37df1f-8c92-46cb-9c89-55cec91d41f9 2024-02-05T05:47:30.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 9c37df1f-8c92-46cb-9c89-55cec91d41f9' 2024-02-05T05:47:30.956 INFO:tasks.workunit.client.0.smithi148.stdout:add osd6 9c37df1f-8c92-46cb-9c89-55cec91d41f9 2024-02-05T05:47:30.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:47:30.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBydsBlVASsOhAAdeZT7+xuBPhTcIo5sg1WOw== 2024-02-05T05:47:30.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBydsBlVASsOhAAdeZT7+xuBPhTcIo5sg1WOw=="}' 2024-02-05T05:47:30.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c37df1f-8c92-46cb-9c89-55cec91d41f9 -i td/osd-scrub-repair/6/new.json 2024-02-05T05:47:31.450 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:47:31.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-02-05T05:47:31.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBydsBlVASsOhAAdeZT7+xuBPhTcIo5sg1WOw== --osd-uuid 9c37df1f-8c92-46cb-9c89-55cec91d41f9 2024-02-05T05:47:31.509 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:47:31.516+0000 7fa929009640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:31.569 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:47:31.576+0000 7fa929009640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:31.570 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:47:31.580+0000 7fa929009640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:31.571 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:47:31.580+0000 7fa929009640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-02-05T05:47:31.571 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:47:31.580+0000 7fa929009640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-02-05T05:47:31.571 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:47:31.580+0000 7fa929009640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-02-05T05:47:31.651 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:47:31.660+0000 7fa929009640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-02-05T05:47:37.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-02-05T05:47:37.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:47:37.858 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd6 key to auth repository 2024-02-05T05:47:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-02-05T05:47:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:47:38.246 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.6 2024-02-05T05:47:38.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-02-05T05:47:38.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:47:38.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:47:38.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:47:38.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:47:38.280 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:47:38.288+0000 7f2e3ce8f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:38.311 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:47:38.320+0000 7f2e3ce8f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:38.312 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:47:38.320+0000 7f2e3ce8f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:38.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-02-05T05:47:38.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:47:38.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-02-05T05:47:38.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:47:38.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:47:38.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:38.618 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:47:38.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:47:38.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:38.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:38.860 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:47:38.868+0000 7f2e3ce8f640 -1 Falling back to public interface 2024-02-05T05:47:38.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:39.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:39.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:39.944 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:47:39.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:47:39.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:39.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:40.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:41.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:41.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:41.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:47:41.256 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:47:41.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:41.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:41.448 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:47:41.455+0000 7f2e3ce8f640 -1 osd.6 0 log_to_monitors true 2024-02-05T05:47:41.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:42.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:42.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:42.571 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:47:42.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:47:42.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:42.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:42.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:43.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:43.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:43.889 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:47:43.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:47:43.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:43.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:44.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:45.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:45.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:45.306 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:47:45.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:47:45.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:45.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:45.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:46.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:46.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:46.626 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:47:46.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:47:46.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:46.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:46.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:47.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:47.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:47.942 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:47:47.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:47:47.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:47.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:48.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:49.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:49.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:49.340 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:47:49.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:47:49.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:49.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:49.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:50.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:50.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:50.659 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:47:50.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:47:50.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:50.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:51.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:52.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:52.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:52.134 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:47:52.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:47:52.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:52.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:52.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:53.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:53.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:53.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:47:53.497 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:47:53.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:53.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:53.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:54.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:54.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:54.806 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:47:54.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:47:54.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:54.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:55.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:56.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:56.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:56.650 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:47:56.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:47:56.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:56.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:57.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:58.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:58.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:58.266 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:47:58.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:47:58.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:58.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:47:58.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:59.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:59.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:59.833 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:47:59.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:47:59.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:59.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:48:00.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:01.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:01.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:01.321 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:48:01.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:48:01.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:01.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:48:01.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:02.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:02.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:02.839 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:48:02.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:48:02.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:02.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:48:03.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:04.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:04.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:04.496 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:48:04.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:48:04.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:04.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:48:04.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:05.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:05.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:05.960 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:48:05.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:48:05.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:05.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:48:06.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:07.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:07.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:07.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:48:07.422 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:48:07.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:07.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:48:07.743 INFO:tasks.workunit.client.0.smithi148.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/909446419,v1:127.0.0.1:6851/909446419] [v2:127.0.0.1:6852/909446419,v1:127.0.0.1:6853/909446419] exists,up 9c37df1f-8c92-46cb-9c89-55cec91d41f9 2024-02-05T05:48:07.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:48:07.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:48:07.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:48:07.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:48:07.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-02-05T05:48:07.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:48:07.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:48:07.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-02-05T05:48:07.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:48:07.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-02-05T05:48:07.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:48:07.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:48:07.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:48:07.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:48:07.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-02-05T05:48:07.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-02-05T05:48:07.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:48:07.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:48:07.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:48:07.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:48:07.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:48:07.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:48:07.749 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:48:07.749 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:48:07.749 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:48:07.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:48:07.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:48:07.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:48:07.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:48:07.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:48:07.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:48:07.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:48:07.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:48:07.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:48:07.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:48:07.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:48:07.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:48:07.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:48:07.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-02-05T05:48:07.751 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:48:07.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0b6436ff-cc1d-4be0-880f-ce0875aa3ee2 2024-02-05T05:48:07.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 0b6436ff-cc1d-4be0-880f-ce0875aa3ee2' 2024-02-05T05:48:07.752 INFO:tasks.workunit.client.0.smithi148.stdout:add osd7 0b6436ff-cc1d-4be0-880f-ce0875aa3ee2 2024-02-05T05:48:07.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:48:07.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXdsBlrJSiLhAABxgs9H3hSmDxKWkoDXc3iA== 2024-02-05T05:48:07.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXdsBlrJSiLhAABxgs9H3hSmDxKWkoDXc3iA=="}' 2024-02-05T05:48:07.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0b6436ff-cc1d-4be0-880f-ce0875aa3ee2 -i td/osd-scrub-repair/7/new.json 2024-02-05T05:48:08.194 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:48:08.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-02-05T05:48:08.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCXdsBlrJSiLhAABxgs9H3hSmDxKWkoDXc3iA== --osd-uuid 0b6436ff-cc1d-4be0-880f-ce0875aa3ee2 2024-02-05T05:48:08.252 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:08.259+0000 7f836ef57640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:08.262 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:08.271+0000 7f836ef57640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:08.263 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:08.271+0000 7f836ef57640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:08.264 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:08.271+0000 7f836ef57640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-02-05T05:48:08.264 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:08.271+0000 7f836ef57640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-02-05T05:48:08.264 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:08.271+0000 7f836ef57640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-02-05T05:48:08.387 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:08.395+0000 7f836ef57640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-02-05T05:48:15.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-02-05T05:48:15.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:48:15.059 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd7 key to auth repository 2024-02-05T05:48:15.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-02-05T05:48:15.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:48:15.445 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.7 2024-02-05T05:48:15.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-02-05T05:48:15.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:48:15.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:48:15.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:48:15.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:48:15.478 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:15.487+0000 7f163daa4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:15.507 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:15.515+0000 7f163daa4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:15.509 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:15.515+0000 7f163daa4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:15.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-02-05T05:48:15.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:48:15.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-02-05T05:48:15.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:48:15.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:48:15.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:15.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:48:15.784 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:48:15.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:15.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:16.054 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:16.063+0000 7f163daa4640 -1 Falling back to public interface 2024-02-05T05:48:16.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:17.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:17.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:17.109 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:48:17.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:48:17.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:17.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:17.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:18.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:18.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:18.424 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:48:18.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:48:18.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:18.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:18.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:18.743 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:18.751+0000 7f163daa4640 -1 osd.7 0 log_to_monitors true 2024-02-05T05:48:19.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:19.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:19.744 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:48:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:48:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:20.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:21.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:21.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:21.145 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:48:21.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:48:21.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:21.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:21.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:22.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:22.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:22.532 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:48:22.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:48:22.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:22.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:22.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:23.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:23.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:23.954 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:48:23.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:48:23.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:23.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:24.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:25.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:25.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:25.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:48:25.271 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:48:25.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:25.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:25.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:26.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:26.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:26.700 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:48:26.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:48:26.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:26.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:27.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:28.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:28.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:28.137 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:48:28.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:48:28.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:28.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:28.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:29.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:29.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:29.460 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:48:29.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:48:29.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:29.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:29.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:30.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:30.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:30.761 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:48:30.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:48:30.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:30.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:31.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:32.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:32.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:32.158 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:48:32.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:48:32.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:32.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:32.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:33.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:33.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:33.474 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:48:33.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:48:33.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:33.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:34.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:35.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:35.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:35.289 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:48:35.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:48:35.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:35.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:35.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:36.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:36.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:36.752 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:48:36.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:48:36.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:36.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:37.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:38.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:38.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:38.383 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:48:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:48:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:38.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:39.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:40.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:40.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:40.317 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:48:40.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:48:40.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:40.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:40.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:41.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:41.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:41.695 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:48:41.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:48:41.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:41.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:42.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:43.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:43.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:43.399 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:48:43.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:48:43.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:43.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:43.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:44.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:44.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:44.728 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:48:44.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:48:44.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:44.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:48:45.050 INFO:tasks.workunit.client.0.smithi148.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1781780411,v1:127.0.0.1:6859/1781780411] [v2:127.0.0.1:6860/1781780411,v1:127.0.0.1:6861/1781780411] exists,up 0b6436ff-cc1d-4be0-880f-ce0875aa3ee2 2024-02-05T05:48:45.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:48:45.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:48:45.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:48:45.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:48:45.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-02-05T05:48:45.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:48:45.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:48:45.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-02-05T05:48:45.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:48:45.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-02-05T05:48:45.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:48:45.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:48:45.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:48:45.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:48:45.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-02-05T05:48:45.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-02-05T05:48:45.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:48:45.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:48:45.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:48:45.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:48:45.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:48:45.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:48:45.055 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:48:45.055 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:48:45.055 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:48:45.055 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:48:45.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:48:45.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:48:45.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:48:45.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:48:45.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:48:45.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:48:45.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:48:45.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:48:45.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:48:45.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:48:45.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:48:45.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:48:45.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-02-05T05:48:45.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:48:45.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e92ce604-e20a-4322-a0c8-40195c87cf4f 2024-02-05T05:48:45.057 INFO:tasks.workunit.client.0.smithi148.stdout:add osd8 e92ce604-e20a-4322-a0c8-40195c87cf4f 2024-02-05T05:48:45.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 e92ce604-e20a-4322-a0c8-40195c87cf4f' 2024-02-05T05:48:45.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:48:45.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9dsBlGk1wBRAAnjQjCurt/7z8oUCsCNWAlg== 2024-02-05T05:48:45.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9dsBlGk1wBRAAnjQjCurt/7z8oUCsCNWAlg=="}' 2024-02-05T05:48:45.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e92ce604-e20a-4322-a0c8-40195c87cf4f -i td/osd-scrub-repair/8/new.json 2024-02-05T05:48:45.454 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:48:45.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-02-05T05:48:45.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9dsBlGk1wBRAAnjQjCurt/7z8oUCsCNWAlg== --osd-uuid e92ce604-e20a-4322-a0c8-40195c87cf4f 2024-02-05T05:48:45.507 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:45.514+0000 7f04f8450640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:45.509 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:45.518+0000 7f04f8450640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:45.511 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:45.518+0000 7f04f8450640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:45.511 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:45.518+0000 7f04f8450640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-02-05T05:48:45.511 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:45.518+0000 7f04f8450640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-02-05T05:48:45.511 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:45.518+0000 7f04f8450640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-02-05T05:48:45.647 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:45.654+0000 7f04f8450640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-02-05T05:48:52.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-02-05T05:48:52.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:48:52.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-02-05T05:48:52.317 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd8 key to auth repository 2024-02-05T05:48:52.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:48:52.705 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.8 2024-02-05T05:48:52.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-02-05T05:48:52.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:48:52.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:48:52.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:48:52.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:48:52.739 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:52.750+0000 7efcba59e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:52.776 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:52.790+0000 7efcba59e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:52.778 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:52.790+0000 7efcba59e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:53.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-02-05T05:48:53.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:48:53.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-02-05T05:48:53.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:48:53.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:48:53.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:53.079 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:48:53.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:48:53.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:53.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:48:53.329 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:53.342+0000 7efcba59e640 -1 Falling back to public interface 2024-02-05T05:48:53.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:54.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:54.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:54.409 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:48:54.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:48:54.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:54.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:48:54.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:55.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:55.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:55.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:48:55.719 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:48:55.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:55.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:48:55.854 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:48:55.866+0000 7efcba59e640 -1 osd.8 0 log_to_monitors true 2024-02-05T05:48:56.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:57.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:57.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:57.089 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:48:57.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:48:57.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:57.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:48:57.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:58.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:58.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:58.643 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:48:58.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:48:58.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:58.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:48:59.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:00.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:00.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:00.017 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:49:00.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:49:00.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:00.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:00.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:01.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:01.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:01.319 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:49:01.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:49:01.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:01.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:01.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:02.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:02.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:02.770 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:49:02.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:49:02.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:02.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:03.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:04.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:04.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:04.117 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:49:04.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:49:04.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:04.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:04.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:05.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:05.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:05.500 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:49:05.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:49:05.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:05.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:05.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:06.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:06.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:06.856 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:49:06.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:49:06.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:06.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:07.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:08.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:08.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:08.174 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:49:08.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:49:08.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:08.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:08.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:09.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:09.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:09.496 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:49:09.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:49:09.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:09.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:09.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:10.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:10.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:10.999 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:49:10.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:49:11.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:11.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:11.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:12.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:12.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:12.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:49:12.322 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:49:12.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:12.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:13.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:14.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:14.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:14.126 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:49:14.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:49:14.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:14.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:14.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:15.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:15.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:15.676 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:49:15.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:49:15.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:15.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:16.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:17.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:17.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:17.259 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:49:17.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:49:17.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:17.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:17.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:18.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:18.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:18.712 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:49:18.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:49:18.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:18.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:19.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:20.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:20.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:20.062 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:49:20.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:49:20.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:20.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:20.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:21.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:21.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:21.548 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:49:21.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:49:21.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:21.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:21.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:22.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:22.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:22.870 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T05:49:22.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T05:49:22.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:22.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:49:23.256 INFO:tasks.workunit.client.0.smithi148.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/1321083950,v1:127.0.0.1:6867/1321083950] [v2:127.0.0.1:6868/1321083950,v1:127.0.0.1:6869/1321083950] exists,up e92ce604-e20a-4322-a0c8-40195c87cf4f 2024-02-05T05:49:23.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:49:23.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:49:23.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:49:23.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:49:23.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-02-05T05:49:23.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:49:23.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:49:23.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-02-05T05:49:23.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:49:23.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:49:23.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:49:23.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:49:23.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:49:23.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:49:23.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:49:23.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:49:23.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:49:23.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:49:23.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:49:23.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:49:23.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:49:23.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:49:23.262 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:49:23.262 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:49:23.262 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:49:23.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:49:23.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:49:23.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:49:23.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:49:23.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:49:23.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:49:23.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:49:23.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:49:23.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:49:23.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:49:23.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:49:23.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:49:23.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:49:23.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:49:23.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:49:23.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d5c58304-d62c-4410-80c8-0849f24f7093 2024-02-05T05:49:23.264 INFO:tasks.workunit.client.0.smithi148.stdout:add osd9 d5c58304-d62c-4410-80c8-0849f24f7093 2024-02-05T05:49:23.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 d5c58304-d62c-4410-80c8-0849f24f7093' 2024-02-05T05:49:23.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:49:23.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDjdsBlrJzSERAAHXjOqb0zsTHDltDyNtA8bw== 2024-02-05T05:49:23.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDjdsBlrJzSERAAHXjOqb0zsTHDltDyNtA8bw=="}' 2024-02-05T05:49:23.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d5c58304-d62c-4410-80c8-0849f24f7093 -i td/osd-scrub-repair/9/new.json 2024-02-05T05:49:23.651 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:49:23.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-02-05T05:49:23.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDjdsBlrJzSERAAHXjOqb0zsTHDltDyNtA8bw== --osd-uuid d5c58304-d62c-4410-80c8-0849f24f7093 2024-02-05T05:49:23.706 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:49:23.717+0000 7f0bd508a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:23.762 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:49:23.773+0000 7f0bd508a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:23.763 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:49:23.777+0000 7f0bd508a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:23.764 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:49:23.777+0000 7f0bd508a640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-02-05T05:49:23.764 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:49:23.777+0000 7f0bd508a640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-02-05T05:49:23.765 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:49:23.777+0000 7f0bd508a640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-02-05T05:49:23.833 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:49:23.845+0000 7f0bd508a640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-02-05T05:49:30.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-02-05T05:49:30.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:49:30.521 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd9 key to auth repository 2024-02-05T05:49:30.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-02-05T05:49:30.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:49:30.901 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.9 2024-02-05T05:49:30.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-02-05T05:49:30.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:49:30.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:49:30.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:49:30.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:49:30.935 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:49:30.949+0000 7fceb3c71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:30.967 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:49:30.977+0000 7fceb3c71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:30.968 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:49:30.981+0000 7fceb3c71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:31.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-02-05T05:49:31.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:49:31.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:49:31.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:49:31.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:49:31.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:31.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:49:31.230 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:49:31.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:31.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:31.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:31.536 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:49:31.549+0000 7fceb3c71640 -1 Falling back to public interface 2024-02-05T05:49:32.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:32.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:32.534 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:49:32.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:49:32.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:32.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:32.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:33.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:33.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:33.858 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:49:33.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:49:33.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:33.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:34.122 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:49:34.133+0000 7fceb3c71640 -1 osd.9 0 log_to_monitors true 2024-02-05T05:49:34.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:35.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:35.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:35.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:49:35.180 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:49:35.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:35.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:35.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:36.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:36.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:36.571 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:49:36.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:49:36.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:36.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:36.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:37.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:37.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:37.953 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:49:37.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:49:37.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:37.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:38.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:39.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:39.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:39.271 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:49:39.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:49:39.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:39.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:39.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:40.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:40.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:40.765 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:49:40.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:49:40.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:40.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:41.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:42.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:42.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:42.182 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:49:42.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:49:42.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:42.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:42.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:43.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:43.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:43.521 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:49:43.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:49:43.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:43.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:44.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:45.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:45.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:45.020 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:49:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:49:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:45.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:46.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:46.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:46.478 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:49:46.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:49:46.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:46.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:46.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:47.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:47.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:47.798 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:49:47.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:49:47.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:47.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:48.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:49.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:49.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:49.116 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:49:49.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:49:49.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:49.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:49.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:50.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:50.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:50.661 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:49:50.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:49:50.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:50.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:51.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:52.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:52.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:52.095 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:49:52.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:49:52.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:52.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:52.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:53.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:53.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:53.518 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:49:53.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:49:53.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:53.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:54.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:55.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:55.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:55.444 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:49:55.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:49:55.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:55.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:56.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:57.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:57.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:57.194 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:49:57.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:49:57.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:57.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:57.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:58.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:58.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:58.959 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:49:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:49:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:49:59.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:00.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:00.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:00.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:50:00.280 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:50:00.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:00.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:50:00.643 INFO:tasks.workunit.client.0.smithi148.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/135217729,v1:127.0.0.1:6875/135217729] [v2:127.0.0.1:6876/135217729,v1:127.0.0.1:6877/135217729] exists,up d5c58304-d62c-4410-80c8-0849f24f7093 2024-02-05T05:50:00.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:50:00.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:50:00.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:50:00.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-02-05T05:50:00.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:50:00.966 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T05:50:00.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:50:00.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:50:01.425 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T05:50:01.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:50:02.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:50:06.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-02-05T05:50:06.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:50:06.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:50:06.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:50:06.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:50:06.082 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:50:06.083 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:50:06.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:50:06.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:50:06.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:50:06.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:50:06.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:50:06.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:50:06.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:50:06.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:50:06.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:50:06.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:50:06.527 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:50:06.527 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:50:06.527 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:50:06.527 INFO:tasks.workunit.client.0.smithi148.stderr:4 2024-02-05T05:50:06.527 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:50:06.528 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:50:06.528 INFO:tasks.workunit.client.0.smithi148.stderr:7 2024-02-05T05:50:06.528 INFO:tasks.workunit.client.0.smithi148.stderr:8 2024-02-05T05:50:06.528 INFO:tasks.workunit.client.0.smithi148.stderr:9' 2024-02-05T05:50:06.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:50:06.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:06.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:50:06.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836553 2024-02-05T05:50:06.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836553 2024-02-05T05:50:06.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553' 2024-02-05T05:50:06.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:06.687 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:50:06.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673033 2024-02-05T05:50:06.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673033 2024-02-05T05:50:06.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673033' 2024-02-05T05:50:06.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:06.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:50:06.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509503 2024-02-05T05:50:06.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509503 2024-02-05T05:50:06.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673033 2-64424509503' 2024-02-05T05:50:06.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:06.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:50:07.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345973 2024-02-05T05:50:07.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345973 2024-02-05T05:50:07.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673033 2-64424509503 3-85899345973' 2024-02-05T05:50:07.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:07.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:50:07.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182448 2024-02-05T05:50:07.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182448 2024-02-05T05:50:07.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673033 2-64424509503 3-85899345973 4-107374182448' 2024-02-05T05:50:07.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:07.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:50:07.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018916 2024-02-05T05:50:07.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018916 2024-02-05T05:50:07.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673033 2-64424509503 3-85899345973 4-107374182448 5-128849018916' 2024-02-05T05:50:07.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:07.386 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:50:07.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855394 2024-02-05T05:50:07.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855394 2024-02-05T05:50:07.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673033 2-64424509503 3-85899345973 4-107374182448 5-128849018916 6-150323855394' 2024-02-05T05:50:07.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:07.527 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:50:07.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691866 2024-02-05T05:50:07.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691866 2024-02-05T05:50:07.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673033 2-64424509503 3-85899345973 4-107374182448 5-128849018916 6-150323855394 7-171798691866' 2024-02-05T05:50:07.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:07.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:50:07.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528335 2024-02-05T05:50:07.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528335 2024-02-05T05:50:07.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673033 2-64424509503 3-85899345973 4-107374182448 5-128849018916 6-150323855394 7-171798691866 8-193273528335' 2024-02-05T05:50:07.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:07.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:50:07.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364805 2024-02-05T05:50:07.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364805 2024-02-05T05:50:07.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673033 2-64424509503 3-85899345973 4-107374182448 5-128849018916 6-150323855394 7-171798691866 8-193273528335 9-214748364805' 2024-02-05T05:50:07.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:07.945 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836553 2024-02-05T05:50:07.945 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:07.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:50:07.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836553 2024-02-05T05:50:07.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:07.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836553 2024-02-05T05:50:07.947 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836553 2024-02-05T05:50:07.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-02-05T05:50:07.947 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:50:08.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836552 -lt 21474836553 2024-02-05T05:50:08.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:50:09.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:50:09.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:50:09.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836554 -lt 21474836553 2024-02-05T05:50:09.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:09.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673033 2024-02-05T05:50:09.555 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:09.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:50:09.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673033 2024-02-05T05:50:09.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:09.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673033 2024-02-05T05:50:09.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673033' 2024-02-05T05:50:09.559 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673033 2024-02-05T05:50:09.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:50:09.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673033 -lt 42949673033 2024-02-05T05:50:09.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:09.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509503 2024-02-05T05:50:09.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:09.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:50:09.906 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509503 2024-02-05T05:50:09.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:09.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509503 2024-02-05T05:50:09.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-02-05T05:50:09.908 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509503 2024-02-05T05:50:09.908 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:50:10.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509503 -lt 64424509503 2024-02-05T05:50:10.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:10.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:10.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345973 2024-02-05T05:50:10.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:50:10.212 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345973 2024-02-05T05:50:10.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:10.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345973 2024-02-05T05:50:10.215 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.3 seq 85899345973 2024-02-05T05:50:10.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345973' 2024-02-05T05:50:10.215 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:50:10.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345974 -lt 85899345973 2024-02-05T05:50:10.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:10.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182448 2024-02-05T05:50:10.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:10.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:50:10.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182448 2024-02-05T05:50:10.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:10.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182448 2024-02-05T05:50:10.535 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.4 seq 107374182448 2024-02-05T05:50:10.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182448' 2024-02-05T05:50:10.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:50:10.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182448 -lt 107374182448 2024-02-05T05:50:10.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:10.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018916 2024-02-05T05:50:10.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:10.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:50:10.845 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018916 2024-02-05T05:50:10.845 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:10.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018916 2024-02-05T05:50:10.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018916' 2024-02-05T05:50:10.846 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.5 seq 128849018916 2024-02-05T05:50:10.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:50:11.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018916 -lt 128849018916 2024-02-05T05:50:11.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:11.162 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855394 2024-02-05T05:50:11.162 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:11.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:50:11.164 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855394 2024-02-05T05:50:11.164 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:11.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855394 2024-02-05T05:50:11.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855394' 2024-02-05T05:50:11.166 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.6 seq 150323855394 2024-02-05T05:50:11.166 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:50:11.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855394 -lt 150323855394 2024-02-05T05:50:11.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:11.491 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691866 2024-02-05T05:50:11.491 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:11.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:50:11.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691866 2024-02-05T05:50:11.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:11.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691866 2024-02-05T05:50:11.495 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.7 seq 171798691866 2024-02-05T05:50:11.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691866' 2024-02-05T05:50:11.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:50:11.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691867 -lt 171798691866 2024-02-05T05:50:11.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:11.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528335 2024-02-05T05:50:11.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:11.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:50:11.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528335 2024-02-05T05:50:11.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:11.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528335 2024-02-05T05:50:11.820 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.8 seq 193273528335 2024-02-05T05:50:11.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528335' 2024-02-05T05:50:11.821 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:50:12.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528335 -lt 193273528335 2024-02-05T05:50:12.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:12.122 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364805 2024-02-05T05:50:12.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:12.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:50:12.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364805 2024-02-05T05:50:12.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:12.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364805 2024-02-05T05:50:12.126 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.9 seq 214748364805 2024-02-05T05:50:12.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364805' 2024-02-05T05:50:12.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:50:12.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364805 -lt 214748364805 2024-02-05T05:50:12.461 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:50:12.462 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:50:12.462 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:50:12.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:50:12.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:50:12.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:50:12.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:50:12.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:50:12.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:50:12.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:50:12.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:50:13.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:50:13.151 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:50:13.151 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:50:13.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:50:13.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:50:13.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:50:13.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:50:13.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-02-05T05:50:13.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:50:13.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:50:13.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-02-05T05:50:13.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:50:13.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-02-05T05:50:13.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:50:13.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:50:14.576 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'ecpool' created 2024-02-05T05:50:14.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:50:15.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-02-05T05:50:15.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-02-05T05:50:16.454 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 2 allow_ec_overwrites to true 2024-02-05T05:50:16.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:50:16.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:50:16.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:50:16.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:50:16.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:50:16.480 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:50:16.480 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:50:16.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:50:16.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:50:16.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:50:16.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:50:16.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:50:16.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:50:16.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:50:16.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:50:16.605 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:50:17.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:50:17.022 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:50:17.022 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:50:17.022 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:50:17.023 INFO:tasks.workunit.client.0.smithi148.stderr:4 2024-02-05T05:50:17.023 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:50:17.023 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:50:17.023 INFO:tasks.workunit.client.0.smithi148.stderr:7 2024-02-05T05:50:17.023 INFO:tasks.workunit.client.0.smithi148.stderr:8 2024-02-05T05:50:17.023 INFO:tasks.workunit.client.0.smithi148.stderr:9' 2024-02-05T05:50:17.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:50:17.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:17.024 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:50:17.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836556 2024-02-05T05:50:17.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836556 2024-02-05T05:50:17.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556' 2024-02-05T05:50:17.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:17.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:50:17.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673036 2024-02-05T05:50:17.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673036 2024-02-05T05:50:17.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673036' 2024-02-05T05:50:17.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:17.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:50:17.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509506 2024-02-05T05:50:17.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509506 2024-02-05T05:50:17.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673036 2-64424509506' 2024-02-05T05:50:17.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:17.648 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:50:17.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345976 2024-02-05T05:50:17.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345976 2024-02-05T05:50:17.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673036 2-64424509506 3-85899345976' 2024-02-05T05:50:17.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:17.786 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:50:17.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182451 2024-02-05T05:50:17.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182451 2024-02-05T05:50:17.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673036 2-64424509506 3-85899345976 4-107374182451' 2024-02-05T05:50:17.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:17.929 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:50:18.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018919 2024-02-05T05:50:18.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018919 2024-02-05T05:50:18.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673036 2-64424509506 3-85899345976 4-107374182451 5-128849018919' 2024-02-05T05:50:18.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:18.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:50:18.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855397 2024-02-05T05:50:18.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855397 2024-02-05T05:50:18.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673036 2-64424509506 3-85899345976 4-107374182451 5-128849018919 6-150323855397' 2024-02-05T05:50:18.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:18.195 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:50:18.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691869 2024-02-05T05:50:18.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691869 2024-02-05T05:50:18.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673036 2-64424509506 3-85899345976 4-107374182451 5-128849018919 6-150323855397 7-171798691869' 2024-02-05T05:50:18.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:18.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:50:18.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528338 2024-02-05T05:50:18.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528338 2024-02-05T05:50:18.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673036 2-64424509506 3-85899345976 4-107374182451 5-128849018919 6-150323855397 7-171798691869 8-193273528338' 2024-02-05T05:50:18.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:18.541 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:50:18.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364808 2024-02-05T05:50:18.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364808 2024-02-05T05:50:18.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673036 2-64424509506 3-85899345976 4-107374182451 5-128849018919 6-150323855397 7-171798691869 8-193273528338 9-214748364808' 2024-02-05T05:50:18.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:18.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836556 2024-02-05T05:50:18.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:18.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:50:18.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836556 2024-02-05T05:50:18.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:18.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836556 2024-02-05T05:50:18.684 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836556 2024-02-05T05:50:18.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-02-05T05:50:18.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:50:18.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836557 -lt 21474836556 2024-02-05T05:50:18.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:18.973 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673036 2024-02-05T05:50:18.973 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:18.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:50:18.975 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673036 2024-02-05T05:50:18.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:18.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673036 2024-02-05T05:50:18.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673036' 2024-02-05T05:50:18.978 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673036 2024-02-05T05:50:18.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:50:19.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673036 -lt 42949673036 2024-02-05T05:50:19.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:19.355 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509506 2024-02-05T05:50:19.355 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:19.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:50:19.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509506 2024-02-05T05:50:19.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:19.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509506 2024-02-05T05:50:19.359 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509506 2024-02-05T05:50:19.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509506' 2024-02-05T05:50:19.359 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:50:19.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509506 -lt 64424509506 2024-02-05T05:50:19.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:19.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:19.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345976 2024-02-05T05:50:19.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:50:19.673 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345976 2024-02-05T05:50:19.673 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:19.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345976 2024-02-05T05:50:19.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345976' 2024-02-05T05:50:19.674 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.3 seq 85899345976 2024-02-05T05:50:19.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:50:19.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345976 -lt 85899345976 2024-02-05T05:50:19.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:19.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182451 2024-02-05T05:50:19.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:19.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:50:19.993 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182451 2024-02-05T05:50:19.994 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:19.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182451 2024-02-05T05:50:19.995 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.4 seq 107374182451 2024-02-05T05:50:19.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2024-02-05T05:50:19.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:50:20.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182451 -lt 107374182451 2024-02-05T05:50:20.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:20.392 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018919 2024-02-05T05:50:20.392 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:20.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:50:20.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018919 2024-02-05T05:50:20.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:20.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018919 2024-02-05T05:50:20.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018919' 2024-02-05T05:50:20.397 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.5 seq 128849018919 2024-02-05T05:50:20.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:50:20.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018919 -lt 128849018919 2024-02-05T05:50:20.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:20.704 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855397 2024-02-05T05:50:20.704 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:20.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:50:20.707 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855397 2024-02-05T05:50:20.707 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:20.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855397 2024-02-05T05:50:20.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855397' 2024-02-05T05:50:20.708 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.6 seq 150323855397 2024-02-05T05:50:20.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:50:21.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855397 -lt 150323855397 2024-02-05T05:50:21.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:21.022 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691869 2024-02-05T05:50:21.022 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:21.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:50:21.025 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691869 2024-02-05T05:50:21.025 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:21.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691869 2024-02-05T05:50:21.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691869' 2024-02-05T05:50:21.027 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.7 seq 171798691869 2024-02-05T05:50:21.027 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:50:21.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691870 -lt 171798691869 2024-02-05T05:50:21.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:21.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528338 2024-02-05T05:50:21.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:21.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:50:21.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528338 2024-02-05T05:50:21.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:21.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528338 2024-02-05T05:50:21.340 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.8 seq 193273528338 2024-02-05T05:50:21.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528338' 2024-02-05T05:50:21.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:50:21.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528338 -lt 193273528338 2024-02-05T05:50:21.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:21.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364808 2024-02-05T05:50:21.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:21.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:50:21.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364808 2024-02-05T05:50:21.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:21.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364808 2024-02-05T05:50:21.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364808' 2024-02-05T05:50:21.697 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.9 seq 214748364808 2024-02-05T05:50:21.697 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:50:22.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364808 -lt 214748364808 2024-02-05T05:50:22.005 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:50:22.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:50:22.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:50:22.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:50:22.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:50:22.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:50:22.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:50:22.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:50:22.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:50:22.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:50:22.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:50:22.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:50:22.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:50:22.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:50:22.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:50:23.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:50:23.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:50:23.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:50:23.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:50:23.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-02-05T05:50:23.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T05:50:23.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-02-05T05:50:23.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-02-05T05:50:23.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:50:23.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:50:23.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T05:50:23.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:50:23.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:50:23.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:50:23.453 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T05:50:23.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:50:24.006 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T05:50:24.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:50:24.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:50:24.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:50:24.650 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-02-05T05:50:24.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T05:50:24.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:50:24.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:50:24.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:50:25.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-02-05T05:50:25.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-02-05T05:50:25.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T05:50:25.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-02-05T05:50:25.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-02-05T05:50:25.102 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:50:25.102 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:50:25.389 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-02-05T05:50:25.389 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:50:25.389 INFO:tasks.workunit.client.0.smithi148.stderr:9 2024-02-05T05:50:25.389 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:50:25.389 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:50:25.389 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:50:25.389 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:50:25.389 INFO:tasks.workunit.client.0.smithi148.stderr:7' 2024-02-05T05:50:25.389 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-02-05T05:50:25.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-02-05T05:50:25.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-02-05T05:50:25.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-02-05T05:50:25.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-02-05T05:50:25.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-02-05T05:50:25.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:50:25.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:50:25.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-02-05T05:50:25.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:50:25.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:50:25.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:50:25.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:50:25.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:50:25.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:50:25.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:50:25.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:50:25.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:50:25.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:50:25.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:50:25.393 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:50:25.393 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:50:25.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:50:25.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:50:25.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:50:25.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:50:25.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:50:25.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:50:25.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:50:25.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:50:25.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:50:25.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:50:25.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:50:27.932 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:50:28.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:50:28.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:50:28.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:50:28.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:50:28.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:50:28.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:50:28.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:50:28.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:50:28.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:50:28.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:50:28.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:50:28.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:50:28.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:50:28.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:50:28.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:50:28.477 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:50:28.477 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:50:28.477 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:50:28.478 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:50:28.478 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:50:28.478 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:50:28.478 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:50:28.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:50:28.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:50:28.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:50:28.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:50:28.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:50:28.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:50:28.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:50:28.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:50:28.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:50:28.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:50:28.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:50:28.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:50:28.480 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.3 2024-02-05T05:50:28.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:50:28.480 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:50:28.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:50:28.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:50:28.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:50:28.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:50:28.512 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:50:28.524+0000 7f137e52c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:50:28.515 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:50:28.528+0000 7f137e52c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:50:28.516 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:50:28.528+0000 7f137e52c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:50:28.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:50:28.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:50:28.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:50:28.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:50:28.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:50:28.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:28.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:50:28.797 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:50:28.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:28.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:50:29.056 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:50:29.068+0000 7f137e52c640 -1 Falling back to public interface 2024-02-05T05:50:29.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:30.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:30.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:30.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:50:30.129 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:50:30.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:30.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:50:30.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:31.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:31.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:31.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:50:31.474 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:50:31.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:31.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:50:31.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:31.791 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:50:31.804+0000 7f137e52c640 -1 osd.3 68 log_to_monitors true 2024-02-05T05:50:32.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:32.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:32.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:50:32.793 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:50:32.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:32.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:50:33.193 INFO:tasks.workunit.client.0.smithi148.stderr:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [20,68) [v2:127.0.0.1:6826/280215854,v1:127.0.0.1:6827/280215854] [v2:127.0.0.1:6828/280215854,v1:127.0.0.1:6829/280215854] exists,up 5498ffc6-6c90-4948-b650-4948ddc04afa 2024-02-05T05:50:33.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:50:33.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:50:33.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:50:33.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:50:33.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:50:33.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:50:33.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:50:33.194 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:50:33.195 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:50:33.195 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:50:33.195 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:50:33.195 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:50:33.195 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:50:33.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:50:33.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:50:33.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:50:33.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:50:33.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:50:33.315 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:50:33.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:50:33.630 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:50:33.630 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:50:33.630 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:50:33.630 INFO:tasks.workunit.client.0.smithi148.stderr:4 2024-02-05T05:50:33.630 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:50:33.630 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:50:33.630 INFO:tasks.workunit.client.0.smithi148.stderr:7 2024-02-05T05:50:33.630 INFO:tasks.workunit.client.0.smithi148.stderr:8 2024-02-05T05:50:33.630 INFO:tasks.workunit.client.0.smithi148.stderr:9' 2024-02-05T05:50:33.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:50:33.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:33.631 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:50:33.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836561 2024-02-05T05:50:33.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836561 2024-02-05T05:50:33.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561' 2024-02-05T05:50:33.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:33.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:50:33.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673040 2024-02-05T05:50:33.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673040 2024-02-05T05:50:33.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673040' 2024-02-05T05:50:33.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:33.953 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:50:34.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509510 2024-02-05T05:50:34.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509510 2024-02-05T05:50:34.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509510' 2024-02-05T05:50:34.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:34.085 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:50:34.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645314 2024-02-05T05:50:34.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645314 2024-02-05T05:50:34.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509510 3-309237645314' 2024-02-05T05:50:34.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:34.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:50:34.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182455 2024-02-05T05:50:34.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182455 2024-02-05T05:50:34.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509510 3-309237645314 4-107374182455' 2024-02-05T05:50:34.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:34.356 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:50:34.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018923 2024-02-05T05:50:34.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018923 2024-02-05T05:50:34.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509510 3-309237645314 4-107374182455 5-128849018923' 2024-02-05T05:50:34.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:34.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:50:34.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855401 2024-02-05T05:50:34.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855401 2024-02-05T05:50:34.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509510 3-309237645314 4-107374182455 5-128849018923 6-150323855401' 2024-02-05T05:50:34.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:34.694 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:50:34.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691874 2024-02-05T05:50:34.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691874 2024-02-05T05:50:34.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509510 3-309237645314 4-107374182455 5-128849018923 6-150323855401 7-171798691874' 2024-02-05T05:50:34.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:34.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:50:34.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528342 2024-02-05T05:50:34.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528342 2024-02-05T05:50:34.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509510 3-309237645314 4-107374182455 5-128849018923 6-150323855401 7-171798691874 8-193273528342' 2024-02-05T05:50:34.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:34.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:50:35.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364812 2024-02-05T05:50:35.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364812 2024-02-05T05:50:35.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509510 3-309237645314 4-107374182455 5-128849018923 6-150323855401 7-171798691874 8-193273528342 9-214748364812' 2024-02-05T05:50:35.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:35.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836561 2024-02-05T05:50:35.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:35.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:50:35.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836561 2024-02-05T05:50:35.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:35.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836561 2024-02-05T05:50:35.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2024-02-05T05:50:35.136 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836561 2024-02-05T05:50:35.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:50:35.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836561 -lt 21474836561 2024-02-05T05:50:35.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:35.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673040 2024-02-05T05:50:35.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:35.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:50:35.442 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673040 2024-02-05T05:50:35.442 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:35.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673040 2024-02-05T05:50:35.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-02-05T05:50:35.443 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949673040 2024-02-05T05:50:35.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:50:35.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673040 -lt 42949673040 2024-02-05T05:50:35.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:35.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509510 2024-02-05T05:50:35.766 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:35.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:50:35.769 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509510 2024-02-05T05:50:35.769 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:35.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509510 2024-02-05T05:50:35.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-02-05T05:50:35.770 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509510 2024-02-05T05:50:35.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:50:36.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509511 -lt 64424509510 2024-02-05T05:50:36.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:36.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:36.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645314 2024-02-05T05:50:36.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:50:36.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645314 2024-02-05T05:50:36.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:36.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645314 2024-02-05T05:50:36.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645314' 2024-02-05T05:50:36.100 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 309237645314 2024-02-05T05:50:36.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:50:36.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645314 -lt 309237645314 2024-02-05T05:50:36.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:36.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:36.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182455 2024-02-05T05:50:36.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:50:36.483 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182455 2024-02-05T05:50:36.483 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:36.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182455 2024-02-05T05:50:36.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182455' 2024-02-05T05:50:36.484 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.4 seq 107374182455 2024-02-05T05:50:36.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:50:36.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182455 -lt 107374182455 2024-02-05T05:50:36.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:36.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018923 2024-02-05T05:50:36.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:36.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:50:36.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018923 2024-02-05T05:50:36.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:36.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018923 2024-02-05T05:50:36.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018923' 2024-02-05T05:50:36.825 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.5 seq 128849018923 2024-02-05T05:50:36.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:50:37.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018924 -lt 128849018923 2024-02-05T05:50:37.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:37.159 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855401 2024-02-05T05:50:37.159 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:37.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:50:37.162 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855401 2024-02-05T05:50:37.162 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:37.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855401 2024-02-05T05:50:37.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855401' 2024-02-05T05:50:37.164 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.6 seq 150323855401 2024-02-05T05:50:37.164 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:50:37.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855401 -lt 150323855401 2024-02-05T05:50:37.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:37.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691874 2024-02-05T05:50:37.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:37.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:50:37.471 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691874 2024-02-05T05:50:37.471 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:37.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691874 2024-02-05T05:50:37.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691874' 2024-02-05T05:50:37.473 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.7 seq 171798691874 2024-02-05T05:50:37.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:50:37.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691874 -lt 171798691874 2024-02-05T05:50:37.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:37.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:37.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528342 2024-02-05T05:50:37.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:50:37.833 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528342 2024-02-05T05:50:37.833 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:37.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528342 2024-02-05T05:50:37.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528342' 2024-02-05T05:50:37.834 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.8 seq 193273528342 2024-02-05T05:50:37.834 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:50:38.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528342 -lt 193273528342 2024-02-05T05:50:38.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:38.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:38.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364812 2024-02-05T05:50:38.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:50:38.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364812 2024-02-05T05:50:38.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:38.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364812 2024-02-05T05:50:38.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364812' 2024-02-05T05:50:38.151 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.9 seq 214748364812 2024-02-05T05:50:38.151 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:50:38.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364813 -lt 214748364812 2024-02-05T05:50:38.478 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:50:38.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:50:38.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:50:38.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:50:38.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:50:38.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:50:38.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:50:38.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:50:38.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:50:38.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:50:38.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:50:39.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:50:39.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:50:39.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:50:39.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:50:39.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:50:39.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:50:39.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:50:39.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:50:39.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:50:39.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:50:39.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:50:39.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:50:39.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:50:39.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:50:39.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:50:39.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:50:39.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:50:39.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:50:39.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:50:39.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:50:40.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:50:14.539466+0000 2024-02-05T05:50:40.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:50:40.637 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:50:40.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:50:14.539466+0000 2024-02-05T05:50:40.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:50:40.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:50:14.539466+0000 2024-02-05T05:50:40.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:50:40.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:50:40.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:50:40.654 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:50:40.655 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:50:40.655 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:50:40.655 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:50:40.655 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:50:40.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:50:14.539466+0000 '>' 2024-02-05T05:50:14.539466+0000 2024-02-05T05:50:40.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:50:41.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:50:41.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:50:41.953 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:50:41.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:50:41.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:50:41.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:50:41.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:50:42.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:50:14.539466+0000 '>' 2024-02-05T05:50:14.539466+0000 2024-02-05T05:50:42.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:50:43.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:50:43.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:50:43.269 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:50:43.269 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:50:43.269 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:50:43.270 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:50:43.270 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:50:43.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:50:40.940846+0000 '>' 2024-02-05T05:50:14.539466+0000 2024-02-05T05:50:43.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:50:43.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:50:43.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:50:43.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:50:43.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:50:43.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:50:43.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:50:43.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:50:43.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:50:43.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:50:43.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:50:43.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:50:43.588 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:50:43.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:50:43.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:50:43.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:50:43.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:50:44.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:50:44.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:50:44.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:50:44.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:50:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:50:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:50:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:50:44.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-02-05T05:50:46.677 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:50:46.677 INFO:tasks.workunit.client.0.smithi148.stdout:hinfo_key 2024-02-05T05:50:46.677 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:50:47.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:50:47.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:50:47.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:50:47.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:50:47.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:50:47.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:50:47.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:50:47.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:50:47.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:50:47.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:50:47.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:50:47.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:50:47.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:50:47.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:50:47.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:50:47.235 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:50:47.235 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:50:47.235 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:50:47.236 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:50:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:50:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:50:47.237 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:50:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:50:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:50:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:50:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:50:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:50:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:50:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:50:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:50:47.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:50:47.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:50:47.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:50:47.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:50:47.238 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.9 2024-02-05T05:50:47.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:50:47.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:50:47.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:50:47.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:50:47.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:50:47.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:50:47.276 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:50:47.287+0000 7f0b019fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:50:47.278 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:50:47.291+0000 7f0b019fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:50:47.280 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:50:47.291+0000 7f0b019fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:50:47.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:50:47.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:50:47.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:50:47.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:50:47.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:50:47.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:47.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:50:47.560 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:50:47.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:50:47.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:47.820 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:50:47.831+0000 7f0b019fc640 -1 Falling back to public interface 2024-02-05T05:50:47.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:48.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:48.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:48.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:50:48.869 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:50:48.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:48.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:50:49.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:50.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:50.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:50.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:50:50.204 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:50:50.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:50.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:50:50.377 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:50:50.391+0000 7f0b019fc640 -1 osd.9 73 log_to_monitors true 2024-02-05T05:50:50.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:51.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:51.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:51.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:50:51.522 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:50:51.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:51.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:50:52.154 INFO:tasks.workunit.client.0.smithi148.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 74 last_clean_interval [50,73) [v2:127.0.0.1:6874/237980271,v1:127.0.0.1:6875/237980271] [v2:127.0.0.1:6876/237980271,v1:127.0.0.1:6877/237980271] exists,up d5c58304-d62c-4410-80c8-0849f24f7093 2024-02-05T05:50:52.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:50:52.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:50:52.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:50:52.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:50:52.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:50:52.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:50:52.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:50:52.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:50:52.155 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:50:52.155 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:50:52.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:50:52.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:50:52.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:50:52.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:50:52.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:50:52.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:50:52.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:50:52.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:50:52.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:50:52.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:50:52.868 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:50:52.868 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:50:52.868 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:50:52.868 INFO:tasks.workunit.client.0.smithi148.stderr:4 2024-02-05T05:50:52.868 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:50:52.868 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:50:52.868 INFO:tasks.workunit.client.0.smithi148.stderr:7 2024-02-05T05:50:52.868 INFO:tasks.workunit.client.0.smithi148.stderr:8 2024-02-05T05:50:52.868 INFO:tasks.workunit.client.0.smithi148.stderr:9' 2024-02-05T05:50:52.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:50:52.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:52.869 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:50:53.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836566 2024-02-05T05:50:53.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836566 2024-02-05T05:50:53.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566' 2024-02-05T05:50:53.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:53.374 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:50:53.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673045 2024-02-05T05:50:53.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673045 2024-02-05T05:50:53.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673045' 2024-02-05T05:50:53.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:53.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:50:53.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509515 2024-02-05T05:50:53.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509515 2024-02-05T05:50:53.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673045 2-64424509515' 2024-02-05T05:50:53.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:53.744 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:50:53.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645319 2024-02-05T05:50:53.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645319 2024-02-05T05:50:53.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673045 2-64424509515 3-309237645319' 2024-02-05T05:50:53.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:53.940 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:50:54.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182460 2024-02-05T05:50:54.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182460 2024-02-05T05:50:54.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673045 2-64424509515 3-309237645319 4-107374182460' 2024-02-05T05:50:54.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:54.085 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:50:54.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018928 2024-02-05T05:50:54.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018928 2024-02-05T05:50:54.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673045 2-64424509515 3-309237645319 4-107374182460 5-128849018928' 2024-02-05T05:50:54.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:54.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:50:54.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855406 2024-02-05T05:50:54.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855406 2024-02-05T05:50:54.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673045 2-64424509515 3-309237645319 4-107374182460 5-128849018928 6-150323855406' 2024-02-05T05:50:54.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:54.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:50:54.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691879 2024-02-05T05:50:54.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691879 2024-02-05T05:50:54.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673045 2-64424509515 3-309237645319 4-107374182460 5-128849018928 6-150323855406 7-171798691879' 2024-02-05T05:50:54.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:54.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:50:54.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528347 2024-02-05T05:50:54.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528347 2024-02-05T05:50:54.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673045 2-64424509515 3-309237645319 4-107374182460 5-128849018928 6-150323855406 7-171798691879 8-193273528347' 2024-02-05T05:50:54.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:54.660 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:50:54.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-02-05T05:50:54.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-02-05T05:50:54.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673045 2-64424509515 3-309237645319 4-107374182460 5-128849018928 6-150323855406 7-171798691879 8-193273528347 9-330712481794' 2024-02-05T05:50:54.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:54.804 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836566 2024-02-05T05:50:54.804 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:54.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:50:54.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836566 2024-02-05T05:50:54.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:54.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836566 2024-02-05T05:50:54.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836566' 2024-02-05T05:50:54.807 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836566 2024-02-05T05:50:54.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:50:55.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836566 -lt 21474836566 2024-02-05T05:50:55.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:55.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673045 2024-02-05T05:50:55.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:55.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:50:55.117 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673045 2024-02-05T05:50:55.117 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:55.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673045 2024-02-05T05:50:55.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-02-05T05:50:55.118 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949673045 2024-02-05T05:50:55.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:50:55.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673045 -lt 42949673045 2024-02-05T05:50:55.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:55.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509515 2024-02-05T05:50:55.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:55.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:50:55.435 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509515 2024-02-05T05:50:55.435 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:55.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509515 2024-02-05T05:50:55.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509515' 2024-02-05T05:50:55.436 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509515 2024-02-05T05:50:55.436 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:50:55.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509516 -lt 64424509515 2024-02-05T05:50:55.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:55.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:55.732 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645319 2024-02-05T05:50:55.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:50:55.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645319 2024-02-05T05:50:55.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:55.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645319 2024-02-05T05:50:55.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645319' 2024-02-05T05:50:55.735 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 309237645319 2024-02-05T05:50:55.735 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:50:56.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645319 -lt 309237645319 2024-02-05T05:50:56.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:56.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182460 2024-02-05T05:50:56.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:56.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:50:56.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182460 2024-02-05T05:50:56.024 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:56.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182460 2024-02-05T05:50:56.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182460' 2024-02-05T05:50:56.025 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.4 seq 107374182460 2024-02-05T05:50:56.025 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:50:56.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182460 -lt 107374182460 2024-02-05T05:50:56.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:56.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018928 2024-02-05T05:50:56.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:56.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:50:56.342 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018928 2024-02-05T05:50:56.343 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:56.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018928 2024-02-05T05:50:56.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018928' 2024-02-05T05:50:56.344 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.5 seq 128849018928 2024-02-05T05:50:56.344 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:50:56.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018929 -lt 128849018928 2024-02-05T05:50:56.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:56.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855406 2024-02-05T05:50:56.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:56.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:50:56.745 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:56.745 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855406 2024-02-05T05:50:56.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855406 2024-02-05T05:50:56.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855406' 2024-02-05T05:50:56.746 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.6 seq 150323855406 2024-02-05T05:50:56.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:50:57.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855406 -lt 150323855406 2024-02-05T05:50:57.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:57.033 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691879 2024-02-05T05:50:57.033 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:57.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:50:57.036 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691879 2024-02-05T05:50:57.036 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:57.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691879 2024-02-05T05:50:57.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-02-05T05:50:57.037 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.7 seq 171798691879 2024-02-05T05:50:57.038 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:50:57.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691879 -lt 171798691879 2024-02-05T05:50:57.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:57.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528347 2024-02-05T05:50:57.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:57.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:50:57.367 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528347 2024-02-05T05:50:57.367 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:57.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528347 2024-02-05T05:50:57.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528347' 2024-02-05T05:50:57.369 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.8 seq 193273528347 2024-02-05T05:50:57.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:50:57.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528347 -lt 193273528347 2024-02-05T05:50:57.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:50:57.700 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481794 2024-02-05T05:50:57.700 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:50:57.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:50:57.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:50:57.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481794 2024-02-05T05:50:57.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-02-05T05:50:57.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-02-05T05:50:57.704 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.9 seq 330712481794 2024-02-05T05:50:57.704 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:50:58.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-02-05T05:50:58.013 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:50:58.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:50:58.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:50:58.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:50:58.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:50:58.374 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:50:58.374 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:50:58.375 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:50:58.375 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:50:58.375 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:50:58.375 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:50:58.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:50:58.654 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:50:58.655 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:50:58.655 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:50:59.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:50:59.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:50:59.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:50:59.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:50:59.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:50:59.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-02-05T05:50:59.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:50:59.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:50:59.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-02-05T05:50:59.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:50:59.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:50:59.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:50:59.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:50:59.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:50:59.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:50:59.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:50:59.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:50:59.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:50:59.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:50:59.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:50:59.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:50:59.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:50:59.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:50:59.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:50:59.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:50:59.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:50:59.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:50:59.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:50:59.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:50:59.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:50:59.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:50:59.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:50:59.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-02-05T05:51:02.202 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:51:02.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:51:02.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:51:02.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:51:02.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:51:02.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:51:02.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:51:02.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:51:02.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:51:02.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:51:02.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:51:02.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:51:02.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:51:02.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:51:02.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:51:02.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:51:02.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:51:02.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:51:02.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:51:02.742 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:51:02.742 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:02.742 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:51:02.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:51:02.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:51:02.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:51:02.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:51:02.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:51:02.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:51:02.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:51:02.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:51:02.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:51:02.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:51:02.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:51:02.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:51:02.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:51:02.744 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.5 2024-02-05T05:51:02.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:51:02.745 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:51:02.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:51:02.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:51:02.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:51:02.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:51:02.777 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:51:02.791+0000 7f83fb65b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:02.779 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:51:02.791+0000 7f83fb65b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:02.781 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:51:02.795+0000 7f83fb65b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:03.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:51:03.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:51:03.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:51:03.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:51:03.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:51:03.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:03.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:51:03.068 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:51:03.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:51:03.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:03.324 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:51:03.335+0000 7f83fb65b640 -1 Falling back to public interface 2024-02-05T05:51:03.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:04.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:04.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:04.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:51:04.389 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:51:04.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:04.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:51:04.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:05.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:05.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:05.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:51:05.708 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:51:05.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:05.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:51:05.826 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:51:05.839+0000 7f83fb65b640 -1 osd.5 79 log_to_monitors true 2024-02-05T05:51:06.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:07.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:07.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:07.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:51:07.112 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:51:07.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:07.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:51:07.587 INFO:tasks.workunit.client.0.smithi148.stderr:osd.5 up in weight 1 up_from 84 up_thru 84 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/2528381221,v1:127.0.0.1:6843/2528381221] [v2:127.0.0.1:6844/2528381221,v1:127.0.0.1:6845/2528381221] exists,up 9832157c-d831-4b53-9b05-71ea506d914b 2024-02-05T05:51:07.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:51:07.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:51:07.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:51:07.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:51:07.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:51:07.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:51:07.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:51:07.588 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:51:07.589 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:51:07.589 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:51:07.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:51:07.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:51:07.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:51:07.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:51:07.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:51:07.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:51:07.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:51:07.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:51:07.696 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:51:08.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:51:08.227 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:51:08.227 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:51:08.227 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:51:08.227 INFO:tasks.workunit.client.0.smithi148.stderr:4 2024-02-05T05:51:08.227 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:51:08.227 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:51:08.227 INFO:tasks.workunit.client.0.smithi148.stderr:7 2024-02-05T05:51:08.227 INFO:tasks.workunit.client.0.smithi148.stderr:8 2024-02-05T05:51:08.227 INFO:tasks.workunit.client.0.smithi148.stderr:9' 2024-02-05T05:51:08.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:51:08.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:08.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:51:08.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836570 2024-02-05T05:51:08.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836570 2024-02-05T05:51:08.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570' 2024-02-05T05:51:08.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:08.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:51:08.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673049 2024-02-05T05:51:08.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673049 2024-02-05T05:51:08.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673049' 2024-02-05T05:51:08.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:08.769 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:51:09.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509519 2024-02-05T05:51:09.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509519 2024-02-05T05:51:09.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673049 2-64424509519' 2024-02-05T05:51:09.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:09.256 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:51:09.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645323 2024-02-05T05:51:09.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645323 2024-02-05T05:51:09.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673049 2-64424509519 3-309237645323' 2024-02-05T05:51:09.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:09.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:51:09.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182464 2024-02-05T05:51:09.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182464 2024-02-05T05:51:09.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673049 2-64424509519 3-309237645323 4-107374182464' 2024-02-05T05:51:09.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:09.657 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:51:09.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-02-05T05:51:09.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-02-05T05:51:09.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673049 2-64424509519 3-309237645323 4-107374182464 5-360777252866' 2024-02-05T05:51:09.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:09.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:51:09.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855410 2024-02-05T05:51:09.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855410 2024-02-05T05:51:09.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673049 2-64424509519 3-309237645323 4-107374182464 5-360777252866 6-150323855410' 2024-02-05T05:51:09.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:09.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:51:10.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691883 2024-02-05T05:51:10.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691883 2024-02-05T05:51:10.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673049 2-64424509519 3-309237645323 4-107374182464 5-360777252866 6-150323855410 7-171798691883' 2024-02-05T05:51:10.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:10.103 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:51:10.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528351 2024-02-05T05:51:10.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528351 2024-02-05T05:51:10.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673049 2-64424509519 3-309237645323 4-107374182464 5-360777252866 6-150323855410 7-171798691883 8-193273528351' 2024-02-05T05:51:10.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:10.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:51:10.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-02-05T05:51:10.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-02-05T05:51:10.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673049 2-64424509519 3-309237645323 4-107374182464 5-360777252866 6-150323855410 7-171798691883 8-193273528351 9-330712481798' 2024-02-05T05:51:10.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:10.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836570 2024-02-05T05:51:10.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:10.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:51:10.388 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836570 2024-02-05T05:51:10.388 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:10.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836570 2024-02-05T05:51:10.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2024-02-05T05:51:10.389 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836570 2024-02-05T05:51:10.390 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:51:10.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836570 -lt 21474836570 2024-02-05T05:51:10.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:10.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673049 2024-02-05T05:51:10.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:10.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:51:10.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673049 2024-02-05T05:51:10.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:10.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673049 2024-02-05T05:51:10.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-02-05T05:51:10.862 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949673049 2024-02-05T05:51:10.862 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:51:11.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673049 -lt 42949673049 2024-02-05T05:51:11.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:11.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:11.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509519 2024-02-05T05:51:11.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:51:11.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509519 2024-02-05T05:51:11.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:11.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509519 2024-02-05T05:51:11.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509519' 2024-02-05T05:51:11.220 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509519 2024-02-05T05:51:11.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:51:11.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509520 -lt 64424509519 2024-02-05T05:51:11.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:11.578 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645323 2024-02-05T05:51:11.579 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:11.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:51:11.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645323 2024-02-05T05:51:11.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:11.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645323 2024-02-05T05:51:11.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645323' 2024-02-05T05:51:11.583 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 309237645323 2024-02-05T05:51:11.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:51:11.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645323 -lt 309237645323 2024-02-05T05:51:11.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:11.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182464 2024-02-05T05:51:11.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:11.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:51:11.884 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182464 2024-02-05T05:51:11.884 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:11.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182464 2024-02-05T05:51:11.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182464' 2024-02-05T05:51:11.886 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.4 seq 107374182464 2024-02-05T05:51:11.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:51:12.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182464 -lt 107374182464 2024-02-05T05:51:12.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:12.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252866 2024-02-05T05:51:12.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:12.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:51:12.173 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252866 2024-02-05T05:51:12.173 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:12.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-02-05T05:51:12.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252866' 2024-02-05T05:51:12.175 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.5 seq 360777252866 2024-02-05T05:51:12.175 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:51:12.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252866 2024-02-05T05:51:12.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:12.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855410 2024-02-05T05:51:12.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:12.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:51:12.545 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855410 2024-02-05T05:51:12.545 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:12.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855410 2024-02-05T05:51:12.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855410' 2024-02-05T05:51:12.546 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.6 seq 150323855410 2024-02-05T05:51:12.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:51:12.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855411 -lt 150323855410 2024-02-05T05:51:12.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:12.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691883 2024-02-05T05:51:12.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:12.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:51:12.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691883 2024-02-05T05:51:12.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:12.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691883 2024-02-05T05:51:12.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-02-05T05:51:12.846 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.7 seq 171798691883 2024-02-05T05:51:12.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:51:13.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691883 -lt 171798691883 2024-02-05T05:51:13.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:13.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528351 2024-02-05T05:51:13.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:13.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:51:13.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528351 2024-02-05T05:51:13.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:13.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528351 2024-02-05T05:51:13.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528351' 2024-02-05T05:51:13.192 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.8 seq 193273528351 2024-02-05T05:51:13.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:51:13.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528351 -lt 193273528351 2024-02-05T05:51:13.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:13.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481798 2024-02-05T05:51:13.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:13.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:51:13.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481798 2024-02-05T05:51:13.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:13.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-02-05T05:51:13.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-02-05T05:51:13.536 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.9 seq 330712481798 2024-02-05T05:51:13.537 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:51:13.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481799 -lt 330712481798 2024-02-05T05:51:13.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:51:13.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:51:13.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:51:14.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:51:14.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:51:14.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:51:14.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:51:14.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:51:14.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:51:14.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:51:14.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:51:14.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:51:14.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:51:14.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:51:14.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:51:14.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:51:14.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:51:14.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:51:14.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:51:14.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:51:14.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:51:14.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:51:14.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:51:15.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:51:15.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:51:15.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:51:15.230 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:51:15.230 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:51:15.231 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:51:15.231 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:51:15.231 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:51:15.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:50:40.940846+0000 2024-02-05T05:51:15.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:51:15.829 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:51:15.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:50:40.940846+0000 2024-02-05T05:51:15.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:51:15.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:50:40.940846+0000 2024-02-05T05:51:15.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:51:15.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:51:15.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:51:15.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:51:15.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:51:15.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:51:15.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:51:15.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:51:16.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:50:40.940846+0000 '>' 2024-02-05T05:50:40.940846+0000 2024-02-05T05:51:16.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:51:17.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:51:17.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:51:17.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:51:17.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:51:17.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:51:17.162 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:51:17.162 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:51:17.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:50:40.940846+0000 '>' 2024-02-05T05:50:40.940846+0000 2024-02-05T05:51:17.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:51:18.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:51:18.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:51:18.480 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:51:18.480 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:51:18.480 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:51:18.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:51:18.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:51:18.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:51:16.066701+0000 '>' 2024-02-05T05:50:40.940846+0000 2024-02-05T05:51:18.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:51:18.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:51:18.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:51:18.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:51:18.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:51:18.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:51:18.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:51:18.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:51:18.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:51:18.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:51:18.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:51:18.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:51:18.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:51:18.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:51:18.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:51:18.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:51:18.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:51:19.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:51:19.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:51:19.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:51:19.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:51:19.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:51:19.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:51:19.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:51:19.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-02-05T05:51:21.733 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:51:21.733 INFO:tasks.workunit.client.0.smithi148.stdout:hinfo_key 2024-02-05T05:51:21.734 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:51:22.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:51:22.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:51:22.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:51:22.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:51:22.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:51:22.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:51:22.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:51:22.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:51:22.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:51:22.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:51:22.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:51:22.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:51:22.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:51:22.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:51:22.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:51:22.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:51:22.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:51:22.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:51:22.283 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:51:22.283 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:22.283 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:51:22.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:51:22.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:51:22.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:51:22.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:51:22.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:51:22.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:51:22.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:51:22.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:51:22.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:51:22.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:51:22.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:51:22.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:51:22.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:51:22.285 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.9 2024-02-05T05:51:22.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:51:22.285 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:51:22.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:51:22.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:51:22.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:51:22.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:51:22.318 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:51:22.330+0000 7f735a211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:22.321 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:51:22.334+0000 7f735a211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:22.323 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:51:22.334+0000 7f735a211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:22.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:51:22.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:51:22.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:51:22.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:51:22.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:51:22.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:22.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:51:22.624 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:51:22.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:22.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:51:22.883 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:51:22.894+0000 7f735a211640 -1 Falling back to public interface 2024-02-05T05:51:22.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:23.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:23.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:23.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:51:23.966 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:51:23.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:23.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:51:24.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:25.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:25.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:25.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:51:25.287 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:51:25.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:25.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:51:25.558 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:51:25.570+0000 7f735a211640 -1 osd.9 87 log_to_monitors true 2024-02-05T05:51:25.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:26.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:26.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:26.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:51:26.607 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:51:26.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:26.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:51:27.124 INFO:tasks.workunit.client.0.smithi148.stderr:osd.9 up in weight 1 up_from 91 up_thru 0 down_at 87 last_clean_interval [77,87) [v2:127.0.0.1:6874/2140320585,v1:127.0.0.1:6875/2140320585] [v2:127.0.0.1:6876/2140320585,v1:127.0.0.1:6877/2140320585] exists,up d5c58304-d62c-4410-80c8-0849f24f7093 2024-02-05T05:51:27.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:51:27.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:51:27.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:51:27.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:51:27.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:51:27.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:51:27.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:51:27.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:51:27.125 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:51:27.125 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:51:27.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:51:27.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:51:27.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:51:27.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:51:27.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:51:27.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:51:27.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:51:27.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:51:27.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:51:27.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:51:27.964 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:51:27.964 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:51:27.965 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:51:27.965 INFO:tasks.workunit.client.0.smithi148.stderr:4 2024-02-05T05:51:27.965 INFO:tasks.workunit.client.0.smithi148.stderr:5 2024-02-05T05:51:27.965 INFO:tasks.workunit.client.0.smithi148.stderr:6 2024-02-05T05:51:27.965 INFO:tasks.workunit.client.0.smithi148.stderr:7 2024-02-05T05:51:27.965 INFO:tasks.workunit.client.0.smithi148.stderr:8 2024-02-05T05:51:27.965 INFO:tasks.workunit.client.0.smithi148.stderr:9' 2024-02-05T05:51:27.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:51:27.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:27.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:51:28.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836575 2024-02-05T05:51:28.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836575 2024-02-05T05:51:28.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575' 2024-02-05T05:51:28.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:28.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:51:28.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673054 2024-02-05T05:51:28.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673054 2024-02-05T05:51:28.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673054' 2024-02-05T05:51:28.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:28.363 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:51:28.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509524 2024-02-05T05:51:28.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509524 2024-02-05T05:51:28.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673054 2-64424509524' 2024-02-05T05:51:28.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:28.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:51:28.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645328 2024-02-05T05:51:28.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645328 2024-02-05T05:51:28.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673054 2-64424509524 3-309237645328' 2024-02-05T05:51:28.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:28.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:51:28.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182469 2024-02-05T05:51:28.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182469 2024-02-05T05:51:28.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673054 2-64424509524 3-309237645328 4-107374182469' 2024-02-05T05:51:28.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:28.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:51:28.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252871 2024-02-05T05:51:28.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252871 2024-02-05T05:51:28.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673054 2-64424509524 3-309237645328 4-107374182469 5-360777252871' 2024-02-05T05:51:28.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:28.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:51:29.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855415 2024-02-05T05:51:29.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855415 2024-02-05T05:51:29.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673054 2-64424509524 3-309237645328 4-107374182469 5-360777252871 6-150323855415' 2024-02-05T05:51:29.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:29.079 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:51:29.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691887 2024-02-05T05:51:29.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691887 2024-02-05T05:51:29.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673054 2-64424509524 3-309237645328 4-107374182469 5-360777252871 6-150323855415 7-171798691887' 2024-02-05T05:51:29.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:29.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:51:29.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528356 2024-02-05T05:51:29.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528356 2024-02-05T05:51:29.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673054 2-64424509524 3-309237645328 4-107374182469 5-360777252871 6-150323855415 7-171798691887 8-193273528356' 2024-02-05T05:51:29.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:29.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:51:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023938 2024-02-05T05:51:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023938 2024-02-05T05:51:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673054 2-64424509524 3-309237645328 4-107374182469 5-360777252871 6-150323855415 7-171798691887 8-193273528356 9-390842023938' 2024-02-05T05:51:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:29.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836575 2024-02-05T05:51:29.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:29.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:51:29.513 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836575 2024-02-05T05:51:29.513 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:29.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836575 2024-02-05T05:51:29.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-02-05T05:51:29.514 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836575 2024-02-05T05:51:29.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:51:29.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836575 -lt 21474836575 2024-02-05T05:51:29.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:29.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:29.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673054 2024-02-05T05:51:29.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:51:29.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673054 2024-02-05T05:51:29.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:29.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673054 2024-02-05T05:51:29.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-02-05T05:51:29.819 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949673054 2024-02-05T05:51:29.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:51:30.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673054 -lt 42949673054 2024-02-05T05:51:30.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:30.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509524 2024-02-05T05:51:30.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:30.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:51:30.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509524 2024-02-05T05:51:30.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:30.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509524 2024-02-05T05:51:30.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-02-05T05:51:30.159 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509524 2024-02-05T05:51:30.159 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:51:30.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509523 -lt 64424509524 2024-02-05T05:51:30.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:51:31.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:51:31.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:51:31.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509524 2024-02-05T05:51:31.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:31.799 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645328 2024-02-05T05:51:31.799 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:31.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:51:31.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645328 2024-02-05T05:51:31.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:31.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645328 2024-02-05T05:51:31.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645328' 2024-02-05T05:51:31.804 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.3 seq 309237645328 2024-02-05T05:51:31.804 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:51:32.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645328 -lt 309237645328 2024-02-05T05:51:32.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:32.113 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182469 2024-02-05T05:51:32.113 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:32.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:51:32.116 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182469 2024-02-05T05:51:32.116 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:32.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182469 2024-02-05T05:51:32.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182469' 2024-02-05T05:51:32.118 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.4 seq 107374182469 2024-02-05T05:51:32.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:51:32.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182469 -lt 107374182469 2024-02-05T05:51:32.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:32.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:32.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252871 2024-02-05T05:51:32.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:51:32.435 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252871 2024-02-05T05:51:32.435 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:32.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252871 2024-02-05T05:51:32.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252871' 2024-02-05T05:51:32.437 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.5 seq 360777252871 2024-02-05T05:51:32.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:51:32.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252872 -lt 360777252871 2024-02-05T05:51:32.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:32.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855415 2024-02-05T05:51:32.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:32.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:51:32.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855415 2024-02-05T05:51:32.783 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:32.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855415 2024-02-05T05:51:32.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855415' 2024-02-05T05:51:32.784 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.6 seq 150323855415 2024-02-05T05:51:32.784 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:51:33.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855416 -lt 150323855415 2024-02-05T05:51:33.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:33.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691887 2024-02-05T05:51:33.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:33.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:51:33.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:33.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691887 2024-02-05T05:51:33.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691887 2024-02-05T05:51:33.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691887' 2024-02-05T05:51:33.113 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.7 seq 171798691887 2024-02-05T05:51:33.113 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:51:33.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691888 -lt 171798691887 2024-02-05T05:51:33.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:33.446 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528356 2024-02-05T05:51:33.446 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:33.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:51:33.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528356 2024-02-05T05:51:33.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:33.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528356 2024-02-05T05:51:33.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528356' 2024-02-05T05:51:33.450 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.8 seq 193273528356 2024-02-05T05:51:33.451 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:51:33.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528356 -lt 193273528356 2024-02-05T05:51:33.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:33.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-390842023938 2024-02-05T05:51:33.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:33.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:51:33.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-390842023938 2024-02-05T05:51:33.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:33.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023938 2024-02-05T05:51:33.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 390842023938' 2024-02-05T05:51:33.796 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.9 seq 390842023938 2024-02-05T05:51:33.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:51:34.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023939 -lt 390842023938 2024-02-05T05:51:34.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:51:34.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:51:34.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:51:34.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:51:34.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:51:34.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:51:34.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:51:34.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:51:34.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:51:34.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:51:34.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:51:34.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:51:34.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:51:34.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:51:34.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:51:35.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:51:35.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:51:35.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:51:35.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:51:35.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:51:35.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-02-05T05:51:35.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:51:35.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:51:35.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-02-05T05:51:35.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-02-05T05:51:35.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:51:35.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:51:35.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:51:35.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:51:35.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 261161"' 2024-02-05T05:51:35.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 261161' 2024-02-05T05:51:35.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:51:35.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:51:35.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:51:35.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 261162"' 2024-02-05T05:51:35.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 261162' 2024-02-05T05:51:35.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:51:35.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 261161 261162' 2024-02-05T05:51:35.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:51:35.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:51:35.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 261161 2024-02-05T05:51:35.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:51:35.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/261164: /' 2024-02-05T05:51:35.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:51:35.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/261166: /' 2024-02-05T05:51:39.413 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:51:39.414 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:51:39.414 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:51:39.414 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:51:39.414 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:51:39.414 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:51:39.414 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:51:39.414 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:51:39.414 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:51:39.414 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:51:39.414 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:51:39.414 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:51:39.415 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:51:39.415 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:51:39.415 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:51:39.415 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:51:39.415 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:51:39.415 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:51:39.415 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:51:39.415 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:51:39.415 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:51:39.415 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:51:39.415 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-02-05T05:51:39.415 INFO:tasks.workunit.client.0.smithi148.stderr:261166: remove 2#2:eb822e21:::SOMETHING:head# 2024-02-05T05:51:39.415 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:51:39.415 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:51:39.416 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:51:39.416 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:51:39.416 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:51:39.416 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:51:39.416 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:51:39.416 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:51:39.419 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:51:39.419 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:51:39.419 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:51:39.419 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:51:39.419 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:51:39.419 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:51:39.419 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:51:39.419 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:51:39.419 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:51:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:51:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:261166: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:51:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:261166: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:261166: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:51:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:51:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:51:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:51:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:51:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:51:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:51:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:51:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:51:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:51:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:51:39.421 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:51:39.421 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:51:39.421 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:51:39.421 INFO:tasks.workunit.client.0.smithi148.stderr:261166: start osd.9 2024-02-05T05:51:39.421 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:51:39.525 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:51:39.525 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:51:39.525 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:51:39.525 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:51:39.525 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:51:39.525 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:51:39.525 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:51:39.525 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:51:39.525 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:51:39.526 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:51:39.526 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:51:39.526 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:51:39.526 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:51:39.526 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:51:39.526 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:51:39.526 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:51:39.526 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:51:39.526 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:51:39.526 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:51:39.526 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:51:39.526 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:51:39.526 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:51:39.526 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-02-05T05:51:39.527 INFO:tasks.workunit.client.0.smithi148.stderr:261164: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:51:39.527 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:51:39.527 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:51:39.527 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:51:39.527 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:51:39.527 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:51:39.527 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:51:39.527 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:51:39.527 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:51:39.529 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:51:39.529 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:51:39.530 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:51:39.530 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:51:39.530 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:51:39.530 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:51:39.530 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:51:39.530 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:51:39.530 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:51:39.530 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:51:39.530 INFO:tasks.workunit.client.0.smithi148.stderr:261164: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:51:39.530 INFO:tasks.workunit.client.0.smithi148.stderr:261164: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:39.530 INFO:tasks.workunit.client.0.smithi148.stderr:261164: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:51:39.530 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:51:39.530 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:51:39.530 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:51:39.531 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:51:39.531 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:51:39.531 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:51:39.531 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:51:39.531 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:51:39.531 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:51:39.531 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:51:39.531 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:51:39.531 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:51:39.531 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:51:39.531 INFO:tasks.workunit.client.0.smithi148.stderr:261164: start osd.5 2024-02-05T05:51:39.531 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:51:44.296 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:51:44.296 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:51:44.296 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:51:44.296 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:51:44.296 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:51:44.296 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 2024-02-05T05:51:39.466+0000 7fd938698640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:44.297 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 2024-02-05T05:51:39.470+0000 7fd938698640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:44.297 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 2024-02-05T05:51:39.470+0000 7fd938698640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:44.297 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:51:44.297 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:51:44.297 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:51:44.297 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:51:44.297 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:51:44.297 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:44.297 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:51:44.297 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 0 2024-02-05T05:51:44.297 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:44.297 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:51:44.297 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 2024-02-05T05:51:40.038+0000 7fd938698640 -1 Falling back to public interface 2024-02-05T05:51:44.297 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:44.298 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:44.298 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:44.298 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:51:44.298 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 1 2024-02-05T05:51:44.298 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:44.298 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:51:44.298 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:44.298 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:44.298 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:44.298 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:51:44.298 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 2 2024-02-05T05:51:44.298 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:51:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 2024-02-05T05:51:42.474+0000 7fd938698640 -1 osd.9 93 log_to_monitors true 2024-02-05T05:51:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:51:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 3 2024-02-05T05:51:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:51:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:261166: osd.9 up in weight 1 up_from 98 up_thru 0 down_at 94 last_c261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:51:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:51:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:51:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:51:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:51:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 2024-02-05T05:51:39.574+0000 7ff07414a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 2024-02-05T05:51:39.578+0000 7ff07414a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 2024-02-05T05:51:39.578+0000 7ff07414a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:51:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:51:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:51:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:51:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:51:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:51:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 0 2024-02-05T05:51:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:51:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 2024-02-05T05:51:40.162+0000 7ff07414a640 -1 Falling back to public interface 2024-02-05T05:51:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:51:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 1 2024-02-05T05:51:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:51:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:51:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 2 2024-02-05T05:51:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:51:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 2024-02-05T05:51:43.614+0000 7ff07414a640 -1 osd.5 93 log_to_monitors true 2024-02-05T05:51:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:51:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 3 2024-02-05T05:51:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:51:45.569 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperlean_interval [91,93) [v2:127.0.0.1:6842/2220015488,v1:127.0.0.1:6843/2220015488] [v2:127.0.0.1:6844/2220015488,v1:127.0.0.1:6845/2220015488] exists,up d5c58304-d62c-4410-80c8-0849f24f7093 2024-02-05T05:51:45.569 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:51:45.569 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:51:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:51:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:51:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:51:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:51:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:51:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:51:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:51:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:51:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:51:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:51:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:51:45.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 1 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 2 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 3 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 4 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 5 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 6 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 7 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 8 2024-02-05T05:51:45.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: 9' 2024-02-05T05:51:45.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:51:45.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:45.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:51:45.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836579 2024-02-05T05:51:45.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836579 2024-02-05T05:51:45.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579' 2024-02-05T05:51:45.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:45.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:51:45.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673058 2024-02-05T05:51:45.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673058 2024-02-05T05:51:45.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673058' 2024-02-05T05:51:45.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:47.025 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qas.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:47.025 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:47.025 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:47.025 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:51:47.025 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 4 2024-02-05T05:51:47.025 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:51:47.025 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:47.025 INFO:tasks.workunit.client.0.smithi148.stderr:261164: osd.5 up in weight 1 up_from 101 up_thru 84 down_at 94 last_clean_interval [84,93) [v2:127.0.0.1:6874/3038573044,v1:127.0.0.1:6875/3038573044] [v2:127.0.0.1:6876/3038573044,v1:127.0.0.1:6877/3038573044] exists,up 9832157c-d831-4b53-9b05-71ea506d914b 2024-02-05T05:51:47.025 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:51:47.025 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:51:47.025 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:51:47.026 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:51:47.026 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:51:47.026 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:51:47.026 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:51:47.026 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:51:47.026 INFO:tasks.workunit.client.0.smithi148.stderr:261164: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:51:47.026 INFO:tasks.workunit.client.0.smithi148.stderr:261164: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:51:47.026 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:51:47.026 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:51:47.026 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:51:47.026 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:51:47.026 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:51:47.026 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:51:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:51:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:51:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:51:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:51:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 1 2024-02-05T05:51:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 2 2024-02-05T05:51:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 3 2024-02-05T05:51:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 4 2024-02-05T05:51:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 5 2024-02-05T05:51:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 6 2024-02-05T05:51:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 7 2024-02-05T05:51:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 8 2024-02-05T05:51:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:261164: 9' 2024-02-05T05:51:47.027 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:51:47.028 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:47.028 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:51:47.028 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836580 2024-02-05T05:51:47.028 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836580 2024-02-05T05:51:47.028 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580' 2024-02-05T05:51:47.028 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:47.861 INFO:tasks.workunit.client.0.smithi148.stderr:261164:/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509529 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509529 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509529' 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645332 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645332 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509529 3-309237645332' 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182473 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182473 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509529 3-309237645332 4-107374182473' 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:47.862 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:51:47.863 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696898 2024-02-05T05:51:47.863 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696898 2024-02-05T05:51:47.863 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509529 3-309237645332 4-107374182473 5-433791696898' 2024-02-05T05:51:47.863 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:47.863 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:51:47.863 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855420 2024-02-05T05:51:47.863 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855420 2024-02-05T05:51:47.863 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509529 3-309237645332 4-107374182473 5-433791696898 6-150323855420' 2024-02-05T05:51:47.863 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:47.863 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:51:47.863 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691892 2024-02-05T05:51:47.863 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691892 2024-02-05T05:51:47.863 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509529 3-309237645332 4-107374182473 5-433791696898 6-150323855420 7-171798691892' 2024-02-05T05:51:47.863 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:47.864 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:51:48.414 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:51:48.414 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673060 2024-02-05T05:51:48.414 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673060 2024-02-05T05:51:48.414 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673060' 2024-02-05T05:51:48.415 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:48.415 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:51:48.415 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509530 2024-02-05T05:51:48.415 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509530 2024-02-05T05:51:48.415 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673060 2-64424509530' 2024-02-05T05:51:48.415 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:48.415 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:51:48.415 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645334 2024-02-05T05:51:48.415 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645334 2024-02-05T05:51:48.415 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673060 2-64424509530 3-309237645334' 2024-02-05T05:51:48.415 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:48.415 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:51:48.415 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182475 2024-02-05T05:51:48.415 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182475 2024-02-05T05:51:48.416 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673060 2-64424509530 3-309237645334 4-107374182475' 2024-02-05T05:51:48.416 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:48.416 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:51:48.416 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696899 2024-02-05T05:51:48.416 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696899 2024-02-05T05:51:48.416 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673060 2-64424509530 3-309237645334 4-107374182475 5-433791696899' 2024-02-05T05:51:48.416 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:48.416 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:51:48.416 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855421 2024-02-05T05:51:48.416 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855421 2024-02-05T05:51:48.416 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673060 2-64424509530 3-309237645334 4-107374182475 5-433791696899 6-150323855421' 2024-02-05T05:51:48.416 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:48.416 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:51:48.602 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flus.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528361 2024-02-05T05:51:48.603 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528361 2024-02-05T05:51:48.603 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509529 3-309237645332 4-107374182473 5-433791696898 6-150323855420 7-171798691892 8-193273528361' 2024-02-05T05:51:48.603 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:48.603 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:51:48.603 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795011 2024-02-05T05:51:48.603 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795011 2024-02-05T05:51:48.603 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509529 3-309237645332 4-107374182473 5-433791696898 6-150323855420 7-171798691892 8-193273528361 9-420906795011' 2024-02-05T05:51:48.603 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:48.603 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836579 2024-02-05T05:51:48.603 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:48.603 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:51:48.604 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836579 2024-02-05T05:51:48.604 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:48.604 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836579 2024-02-05T05:51:48.604 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-02-05T05:51:48.604 INFO:tasks.workunit.client.0.smithi148.stderr:261166: waiting osd.0 seq 21474836579 2024-02-05T05:51:48.604 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:51:48.604 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836579 2024-02-05T05:51:48.604 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:48.604 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673058 2024-02-05T05:51:48.604 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:48.604 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:51:48.604 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673058 2024-02-05T05:51:48.604 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:48.604 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673058 2024-02-05T05:51:48.605 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-02-05T05:51:48.605 INFO:tasks.workunit.client.0.smithi148.stderr:261166: waiting osd.1 seq 42949673058 2024-02-05T05:51:48.605 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:51:48.605 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673060 -lt 42949673058 2024-02-05T05:51:48.605 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:48.605 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509529 2024-02-05T05:51:48.605 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:48.605 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:51:48.973 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.ch_pg_stats: seq=171798691893 2024-02-05T05:51:48.973 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691893 2024-02-05T05:51:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673060 2-64424509530 3-309237645334 4-107374182475 5-433791696899 6-150323855421 7-171798691893' 2024-02-05T05:51:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:51:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528362 2024-02-05T05:51:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528362 2024-02-05T05:51:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673060 2-64424509530 3-309237645334 4-107374182475 5-433791696899 6-150323855421 7-171798691893 8-193273528362' 2024-02-05T05:51:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:51:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795012 2024-02-05T05:51:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795012 2024-02-05T05:51:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673060 2-64424509530 3-309237645334 4-107374182475 5-433791696899 6-150323855421 7-171798691893 8-193273528362 9-420906795012' 2024-02-05T05:51:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836580 2024-02-05T05:51:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:51:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836580 2024-02-05T05:51:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836580 2024-02-05T05:51:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-02-05T05:51:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:261164: waiting osd.0 seq 21474836580 2024-02-05T05:51:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:51:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836581 -lt 21474836580 2024-02-05T05:51:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673060 2024-02-05T05:51:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:51:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673060 2024-02-05T05:51:48.976 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:48.976 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673060 2024-02-05T05:51:48.976 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673060' 2024-02-05T05:51:48.976 INFO:tasks.workunit.client.0.smithi148.stderr:261164: waiting osd.1 seq 42949673060 2024-02-05T05:51:49.568 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pglient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509529 2024-02-05T05:51:49.568 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:49.569 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509529 2024-02-05T05:51:49.569 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-02-05T05:51:49.569 INFO:tasks.workunit.client.0.smithi148.stderr:261166: waiting osd.2 seq 64424509529 2024-02-05T05:51:49.569 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:51:49.569 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509530 -lt 64424509529 2024-02-05T05:51:49.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:49.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645332 2024-02-05T05:51:49.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:49.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:51:49.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645332 2024-02-05T05:51:49.570 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:49.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645332 2024-02-05T05:51:49.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645332' 2024-02-05T05:51:49.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: waiting osd.3 seq 309237645332 2024-02-05T05:51:49.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:51:49.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645334 -lt 309237645332 2024-02-05T05:51:49.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:49.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182473 2024-02-05T05:51:49.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:49.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:51:49.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182473 2024-02-05T05:51:49.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:49.571 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182473 2024-02-05T05:51:49.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182473' 2024-02-05T05:51:49.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: waiting osd.4 seq 107374182473 2024-02-05T05:51:49.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:51:49.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182475 -lt 107374182473 2024-02-05T05:51:49.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:49.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-433791696898 2024-02-05T05:51:49.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:49.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:51:49.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-433791696898 2024-02-05T05:51:49.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:49.572 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696898 2024-02-05T05:51:50.265 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiti_stats: ceph osd last-stat-seq 1 2024-02-05T05:51:50.265 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673060 -lt 42949673060 2024-02-05T05:51:50.265 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:50.265 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509530 2024-02-05T05:51:50.265 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:50.265 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:51:50.265 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509530 2024-02-05T05:51:50.265 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:50.265 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509530 2024-02-05T05:51:50.266 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-02-05T05:51:50.266 INFO:tasks.workunit.client.0.smithi148.stderr:261164: waiting osd.2 seq 64424509530 2024-02-05T05:51:50.266 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:51:50.266 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509530 -lt 64424509530 2024-02-05T05:51:50.266 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:50.266 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645334 2024-02-05T05:51:50.266 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:50.266 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:51:50.266 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645334 2024-02-05T05:51:50.266 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:50.266 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645334 2024-02-05T05:51:50.266 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645334' 2024-02-05T05:51:50.266 INFO:tasks.workunit.client.0.smithi148.stderr:261164: waiting osd.3 seq 309237645334 2024-02-05T05:51:50.266 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:51:50.267 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645334 -lt 309237645334 2024-02-05T05:51:50.267 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:50.267 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182475 2024-02-05T05:51:50.267 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:50.267 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:51:50.267 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182475 2024-02-05T05:51:50.267 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:50.267 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182475 2024-02-05T05:51:50.267 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182475' 2024-02-05T05:51:50.267 INFO:tasks.workunit.client.0.smithi148.stderr:261164: waiting osd.4 seq 107374182475 2024-02-05T05:51:50.267 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:51:50.267 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182475 -lt 107374182475 2024-02-05T05:51:50.267 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:50.267 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-433791696899 2024-02-05T05:51:50.898 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/ng osd.5 seq 433791696898' 2024-02-05T05:51:50.899 INFO:tasks.workunit.client.0.smithi148.stderr:261166: waiting osd.5 seq 433791696898 2024-02-05T05:51:50.899 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:51:50.899 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696899 -lt 433791696898 2024-02-05T05:51:50.899 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:50.899 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855420 2024-02-05T05:51:50.899 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:50.899 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:51:50.899 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855420 2024-02-05T05:51:50.899 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:50.899 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855420 2024-02-05T05:51:50.899 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855420' 2024-02-05T05:51:50.900 INFO:tasks.workunit.client.0.smithi148.stderr:261166: waiting osd.6 seq 150323855420 2024-02-05T05:51:50.900 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:51:50.900 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855421 -lt 150323855420 2024-02-05T05:51:50.900 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:50.900 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:50.900 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691892 2024-02-05T05:51:50.900 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:51:50.900 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691892 2024-02-05T05:51:50.900 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:50.900 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691892 2024-02-05T05:51:50.900 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691892' 2024-02-05T05:51:50.900 INFO:tasks.workunit.client.0.smithi148.stderr:261166: waiting osd.7 seq 171798691892 2024-02-05T05:51:50.900 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:51:50.901 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691894 -lt 171798691892 2024-02-05T05:51:50.901 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:50.901 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528361 2024-02-05T05:51:50.901 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:50.901 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:51:50.901 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528361 2024-02-05T05:51:50.901 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:50.901 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528361 2024-02-05T05:51:50.901 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528361' 2024-02-05T05:51:50.901 INFO:tasks.workunit.client.0.smithi148.stderr:261166: waiting osd.8 seq 193273528361 2024-02-05T05:51:50.901 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:51:50.901 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528362 -lt 193273528361 2024-02-05T05:51:51.224 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pgcephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:51.224 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:51:51.224 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-433791696899 2024-02-05T05:51:51.224 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:51.224 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696899 2024-02-05T05:51:51.224 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 433791696899' 2024-02-05T05:51:51.224 INFO:tasks.workunit.client.0.smithi148.stderr:261164: waiting osd.5 seq 433791696899 2024-02-05T05:51:51.224 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:51:51.224 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696900 -lt 433791696899 2024-02-05T05:51:51.224 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:51.224 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855421 2024-02-05T05:51:51.225 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:51.225 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:51:51.225 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855421 2024-02-05T05:51:51.225 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:51.225 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855421 2024-02-05T05:51:51.225 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855421' 2024-02-05T05:51:51.225 INFO:tasks.workunit.client.0.smithi148.stderr:261164: waiting osd.6 seq 150323855421 2024-02-05T05:51:51.225 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:51:51.225 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855421 -lt 150323855421 2024-02-05T05:51:51.225 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:51.225 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691893 2024-02-05T05:51:51.225 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:51.225 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:51:51.225 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691893 2024-02-05T05:51:51.226 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:51.226 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691893 2024-02-05T05:51:51.226 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691893' 2024-02-05T05:51:51.226 INFO:tasks.workunit.client.0.smithi148.stderr:261164: waiting osd.7 seq 171798691893 2024-02-05T05:51:51.226 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:51:51.226 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691894 -lt 171798691893 2024-02-05T05:51:51.226 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:51.226 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:51.226 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528362 2024-02-05T05:51:51.226 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:51:51.226 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528362 2024-02-05T05:51:52.356 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: _stats: for s in $seqs 2024-02-05T05:51:52.356 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-420906795011 2024-02-05T05:51:52.356 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:52.356 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:51:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-420906795011 2024-02-05T05:51:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795011 2024-02-05T05:51:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 420906795011' 2024-02-05T05:51:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:261166: waiting osd.9 seq 420906795011 2024-02-05T05:51:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:51:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795012 -lt 420906795011 2024-02-05T05:51:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:51:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:51:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:51:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:51:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:51:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:51:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:51:52.358 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:51:52.358 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:51:52.358 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:51:52.358 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:51:52.358 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:51:52.358 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:51:52.358 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:51:52.358 INFO:tasks.workunit.client.0.smithi148.stderr:261166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:51:52.358 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:51:52.358 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:51:52.358 INFO:tasks.workunit.client.0.smithi148.stderr:261166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:51:52.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:51:52.941 INFO:tasks.workunit.client.0.smithi148.stderr: cut -d - -f 2 2024-02-05T05:51:52.942 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528362 2024-02-05T05:51:52.942 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528362' 2024-02-05T05:51:52.942 INFO:tasks.workunit.client.0.smithi148.stderr:261164: waiting osd.8 seq 193273528362 2024-02-05T05:51:52.942 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:51:52.942 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528362 -lt 193273528362 2024-02-05T05:51:52.942 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:52.942 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-420906795012 2024-02-05T05:51:52.942 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:52.942 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:51:52.942 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-420906795012 2024-02-05T05:51:52.942 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:52.942 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795012 2024-02-05T05:51:52.943 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 420906795012' 2024-02-05T05:51:52.943 INFO:tasks.workunit.client.0.smithi148.stderr:261164: waiting osd.9 seq 420906795012 2024-02-05T05:51:52.943 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:51:52.943 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795012 -lt 420906795012 2024-02-05T05:51:52.943 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:51:52.943 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:51:52.943 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:51:52.943 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:51:52.943 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:51:52.943 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:51:52.943 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:51:52.943 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:51:52.943 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:51:52.943 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:51:52.944 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:51:52.944 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:51:52.944 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:51:52.944 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:51:52.944 INFO:tasks.workunit.client.0.smithi148.stderr:261164: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:51:52.944 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:51:52.944 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:51:52.944 INFO:tasks.workunit.client.0.smithi148.stderr:261164: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:51:52.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:51:52.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:51:52.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 261162 2024-02-05T05:51:52.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:51:52.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:51:52.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:51:52.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:51:52.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:51:52.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:51:52.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:51:52.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:51:52.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:51:52.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:51:53.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:51:53.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:51:53.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:51:53.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:51:53.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:51:53.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:51:53.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:51:53.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:51:53.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:51:16.066701+0000 2024-02-05T05:51:53.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:51:53.907 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:51:53.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:51:16.066701+0000 2024-02-05T05:51:53.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:51:53.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:51:16.066701+0000 2024-02-05T05:51:53.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:51:53.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:51:53.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:51:53.925 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:51:53.925 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:51:53.925 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:51:53.925 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:51:53.925 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:51:54.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:51:16.066701+0000 '>' 2024-02-05T05:51:16.066701+0000 2024-02-05T05:51:54.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:51:55.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:51:55.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:51:55.231 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:51:55.231 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:51:55.231 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:51:55.231 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:51:55.232 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:51:55.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:51:16.066701+0000 '>' 2024-02-05T05:51:16.066701+0000 2024-02-05T05:51:55.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:51:56.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:51:56.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:51:56.549 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:51:56.549 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:51:56.549 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:51:56.550 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:51:56.550 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:51:56.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:51:16.066701+0000 '>' 2024-02-05T05:51:16.066701+0000 2024-02-05T05:51:56.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:51:57.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:51:57.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:51:57.869 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:51:57.869 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:51:57.869 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:51:57.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:51:57.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:51:58.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:51:16.066701+0000 '>' 2024-02-05T05:51:16.066701+0000 2024-02-05T05:51:58.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:51:59.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:51:59.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:51:59.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:51:59.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:51:59.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:51:59.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:51:59.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:51:59.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:51:54.839405+0000 '>' 2024-02-05T05:51:16.066701+0000 2024-02-05T05:51:59.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:51:59.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:51:59.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:51:59.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:51:59.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:51:59.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 264593"' 2024-02-05T05:51:59.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 264593' 2024-02-05T05:51:59.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:51:59.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:51:59.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:51:59.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 264594"' 2024-02-05T05:51:59.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 264594' 2024-02-05T05:51:59.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:51:59.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 264593 264594' 2024-02-05T05:51:59.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:51:59.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:51:59.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 264593 2024-02-05T05:51:59.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:51:59.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/264596: /' 2024-02-05T05:51:59.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:51:59.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/264598: /' 2024-02-05T05:52:02.890 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:52:02.891 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:52:02.891 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:52:02.891 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:52:02.891 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:52:02.891 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:52:02.891 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:52:02.891 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:52:02.891 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:52:02.891 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:52:02.891 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:52:02.891 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:52:02.891 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:52:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:52:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:52:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:52:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:52:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:52:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:52:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:52:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:52:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:52:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-02-05T05:52:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:264598: _ 2024-02-05T05:52:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:264598: hinfo_key 2024-02-05T05:52:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:264598: snapset 2024-02-05T05:52:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:52:02.893 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:52:02.893 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:52:02.893 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:52:02.893 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:52:02.893 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:52:02.893 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:52:02.893 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:52:02.895 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:52:02.895 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:52:02.895 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:52:02.895 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:52:02.895 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:52:02.895 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:52:02.895 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:52:02.895 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:52:02.895 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:52:02.895 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:52:02.896 INFO:tasks.workunit.client.0.smithi148.stderr:264598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:52:02.896 INFO:tasks.workunit.client.0.smithi148.stderr:264598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:02.896 INFO:tasks.workunit.client.0.smithi148.stderr:264598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:52:02.896 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:02.896 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:02.896 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:52:02.896 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:52:02.896 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:52:02.896 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:52:02.896 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:52:02.896 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:52:02.896 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:52:02.896 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:52:02.896 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:52:02.897 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:52:02.897 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:52:02.897 INFO:tasks.workunit.client.0.smithi148.stderr:264598: start osd.9 2024-02-05T05:52:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:52:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:52:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:52:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:52:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:52:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:52:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:52:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:52:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:52:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:52:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:52:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:52:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:52:03.373 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:52:03.373 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:52:03.373 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:52:03.373 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:52:03.373 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:52:03.373 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:52:03.373 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:52:03.373 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:52:03.373 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:52:03.373 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-02-05T05:52:03.373 INFO:tasks.workunit.client.0.smithi148.stderr:264596: _ 2024-02-05T05:52:03.373 INFO:tasks.workunit.client.0.smithi148.stderr:264596: hinfo_key 2024-02-05T05:52:03.373 INFO:tasks.workunit.client.0.smithi148.stderr:264596: snapset 2024-02-05T05:52:03.373 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:52:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:52:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:52:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:52:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:52:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:52:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:52:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:52:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:52:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:52:03.374 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:52:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:52:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:52:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:52:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:52:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:52:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:52:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:52:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:264596: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:52:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:264596: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:264596: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:52:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:52:03.375 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:52:03.376 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:52:03.376 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:52:03.376 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:52:03.376 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:52:03.376 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:52:03.376 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:52:03.376 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:52:03.376 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:52:03.376 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:52:03.376 INFO:tasks.workunit.client.0.smithi148.stderr:264596: start osd.5 2024-02-05T05:52:07.480 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:52:07.480 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:52:07.480 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:52:07.480 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:52:07.480 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:52:07.480 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:52:07.481 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 2024-02-05T05:52:02.945+0000 7f073f4b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:07.481 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 2024-02-05T05:52:02.945+0000 7f073f4b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:07.481 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 2024-02-05T05:52:02.949+0000 7f073f4b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:07.481 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:52:07.481 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:52:07.481 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:52:07.481 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:52:07.481 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:52:07.481 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:07.481 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:52:07.481 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 0 2024-02-05T05:52:07.482 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:07.482 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:52:07.482 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 2024-02-05T05:52:03.513+0000 7f073f4b8640 -1 Falling back to public interface 2024-02-05T05:52:07.482 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:07.482 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:07.482 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:07.482 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:52:07.482 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 1 2024-02-05T05:52:07.483 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:07.483 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:52:07.483 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:07.483 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:07.483 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:07.483 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:52:07.483 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 2 2024-02-05T05:52:07.483 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:07.483 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:52:07.483 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 2024-02-05T05:52:06.017+0000 7f073f4b8640 -1 osd.9 103 log_to_monitors true 2024-02-05T05:52:07.483 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:07.484 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:07.484 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:07.484 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:52:07.484 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 3 2024-02-05T05:52:07.484 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:07.484 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:52:08.489 INFO:tasks.workunit.client.0.smithi148.stderr:264598: osd.9 up in weight 1 up_from 108 up_thru 0 down_overy_ops 2024-02-05T05:52:08.489 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:52:08.489 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:52:08.489 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:52:08.489 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:52:08.489 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:52:08.489 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 2024-02-05T05:52:03.433+0000 7ff68e5cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:08.489 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 2024-02-05T05:52:03.469+0000 7ff68e5cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:08.489 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 2024-02-05T05:52:03.469+0000 7ff68e5cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:08.489 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:52:08.489 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:52:08.490 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:52:08.490 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:52:08.490 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:52:08.490 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:08.490 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:52:08.490 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 0 2024-02-05T05:52:08.490 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:08.490 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:52:08.490 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 2024-02-05T05:52:04.013+0000 7ff68e5cd640 -1 Falling back to public interface 2024-02-05T05:52:08.490 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:08.490 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:08.490 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:08.490 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:52:08.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 1 2024-02-05T05:52:08.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:08.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:52:08.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:08.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:08.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:08.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:52:08.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 2 2024-02-05T05:52:08.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:08.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:52:08.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:08.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 2024-02-05T05:52:07.165+0000 7ff68e5cd640 -1 osd.5 103 log_to_monitors true 2024-02-05T05:52:08.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:08.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:08.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:52:08.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 3 2024-02-05T05:52:08.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:08.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:52:08.754 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/at 104 last_clean_interval [98,103) [v2:127.0.0.1:6842/1493911545,v1:127.0.0.1:6843/1493911545] [v2:127.0.0.1:6844/1493911545,v1:127.0.0.1:6845/1493911545] exists,up d5c58304-d62c-4410-80c8-0849f24f7093 2024-02-05T05:52:08.754 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:52:08.755 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:52:08.755 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:52:08.755 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:52:08.755 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:52:08.755 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:52:08.755 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:52:08.755 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:52:08.755 INFO:tasks.workunit.client.0.smithi148.stderr:264598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:52:08.755 INFO:tasks.workunit.client.0.smithi148.stderr:264598: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:52:08.755 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:52:08.755 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:52:08.756 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:52:08.756 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:52:08.756 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:52:08.756 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:52:08.756 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:52:08.756 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:52:08.756 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:52:08.756 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:52:08.756 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 1 2024-02-05T05:52:08.756 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 2 2024-02-05T05:52:08.756 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 3 2024-02-05T05:52:08.757 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 4 2024-02-05T05:52:08.757 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 5 2024-02-05T05:52:08.757 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 6 2024-02-05T05:52:08.757 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 7 2024-02-05T05:52:08.757 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 8 2024-02-05T05:52:08.757 INFO:tasks.workunit.client.0.smithi148.stderr:264598: 9' 2024-02-05T05:52:08.757 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:52:08.757 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:08.757 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:52:08.757 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836586 2024-02-05T05:52:08.757 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836586 2024-02-05T05:52:08.757 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586' 2024-02-05T05:52:08.758 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:08.758 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:52:08.758 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673065 2024-02-05T05:52:08.758 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673065 2024-02-05T05:52:08.758 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673065' 2024-02-05T05:52:08.758 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:10.942 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/cloceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:10.942 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:10.942 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:10.942 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:52:10.942 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 4 2024-02-05T05:52:10.942 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:10.942 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:52:10.942 INFO:tasks.workunit.client.0.smithi148.stderr:264596: osd.5 up in weight 1 up_from 111 up_thru 101 down_at 104 last_clean_interval [101,103) [v2:127.0.0.1:6874/382512330,v1:127.0.0.1:6875/382512330] [v2:127.0.0.1:6876/382512330,v1:127.0.0.1:6877/382512330] exists,up 9832157c-d831-4b53-9b05-71ea506d914b 2024-02-05T05:52:10.942 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:52:10.942 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:52:10.942 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:52:10.943 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:52:10.943 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:52:10.943 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:52:10.943 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:52:10.943 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:52:10.943 INFO:tasks.workunit.client.0.smithi148.stderr:264596: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:52:10.943 INFO:tasks.workunit.client.0.smithi148.stderr:264596: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:52:10.943 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:52:10.943 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:52:10.943 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:52:10.943 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:52:10.943 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:52:10.943 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:52:10.944 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:52:10.944 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:52:10.944 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:52:10.944 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:52:10.944 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 1 2024-02-05T05:52:10.944 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 2 2024-02-05T05:52:10.945 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 3 2024-02-05T05:52:10.945 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 4 2024-02-05T05:52:10.945 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 5 2024-02-05T05:52:10.945 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 6 2024-02-05T05:52:10.945 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 7 2024-02-05T05:52:10.945 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 8 2024-02-05T05:52:10.946 INFO:tasks.workunit.client.0.smithi148.stderr:264596: 9' 2024-02-05T05:52:10.946 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:52:10.946 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:10.946 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:52:10.946 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836587 2024-02-05T05:52:10.946 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836587 2024-02-05T05:52:10.946 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587' 2024-02-05T05:52:11.460 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ne.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:52:11.460 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509535 2024-02-05T05:52:11.460 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509535 2024-02-05T05:52:11.460 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673065 2-64424509535' 2024-02-05T05:52:11.461 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:11.461 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:52:11.461 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645339 2024-02-05T05:52:11.461 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645339 2024-02-05T05:52:11.461 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673065 2-64424509535 3-309237645339' 2024-02-05T05:52:11.461 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:11.461 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:52:11.461 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182480 2024-02-05T05:52:11.461 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182480 2024-02-05T05:52:11.461 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673065 2-64424509535 3-309237645339 4-107374182480' 2024-02-05T05:52:11.461 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:11.461 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:52:11.461 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369858 2024-02-05T05:52:11.461 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369858 2024-02-05T05:52:11.462 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673065 2-64424509535 3-309237645339 4-107374182480 5-476741369858' 2024-02-05T05:52:11.462 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:11.462 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:52:11.462 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855426 2024-02-05T05:52:11.462 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855426 2024-02-05T05:52:11.462 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673065 2-64424509535 3-309237645339 4-107374182480 5-476741369858 6-150323855426' 2024-02-05T05:52:11.462 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:11.462 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:52:11.462 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691899 2024-02-05T05:52:11.462 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691899 2024-02-05T05:52:11.462 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673065 2-64424509535 3-309237645339 4-107374182480 5-476741369858 6-150323855426 7-171798691899' 2024-02-05T05:52:11.462 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:11.462 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:52:12.361 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/ids 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673067 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673067 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673067' 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509537 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509537 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673067 2-64424509537' 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645340 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645340 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673067 2-64424509537 3-309237645340' 2024-02-05T05:52:12.362 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:12.363 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:52:12.363 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182481 2024-02-05T05:52:12.363 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182481 2024-02-05T05:52:12.363 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673067 2-64424509537 3-309237645340 4-107374182481' 2024-02-05T05:52:12.363 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:12.363 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:52:12.363 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369859 2024-02-05T05:52:12.363 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369859 2024-02-05T05:52:12.363 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673067 2-64424509537 3-309237645340 4-107374182481 5-476741369859' 2024-02-05T05:52:12.363 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:12.363 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:52:12.363 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855428 2024-02-05T05:52:12.363 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855428 2024-02-05T05:52:12.364 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673067 2-64424509537 3-309237645340 4-107374182481 5-476741369859 6-150323855428' 2024-02-05T05:52:12.364 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:12.364 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:52:12.389 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shcephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528367 2024-02-05T05:52:12.389 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528367 2024-02-05T05:52:12.389 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673065 2-64424509535 3-309237645339 4-107374182480 5-476741369858 6-150323855426 7-171798691899 8-193273528367' 2024-02-05T05:52:12.389 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467971 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467971 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673065 2-64424509535 3-309237645339 4-107374182480 5-476741369858 6-150323855426 7-171798691899 8-193273528367 9-463856467971' 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836586 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836586 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836586 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: waiting osd.0 seq 21474836586 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:52:12.390 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836586 -lt 21474836586 2024-02-05T05:52:12.391 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:12.391 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673065 2024-02-05T05:52:12.391 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:12.391 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:52:12.391 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673065 2024-02-05T05:52:12.391 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:12.391 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673065 2024-02-05T05:52:12.391 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673065' 2024-02-05T05:52:12.391 INFO:tasks.workunit.client.0.smithi148.stderr:264598: waiting osd.1 seq 42949673065 2024-02-05T05:52:12.391 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:52:12.391 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673065 -lt 42949673065 2024-02-05T05:52:12.391 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:12.391 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509535 2024-02-05T05:52:12.392 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:12.392 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:52:13.130 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/ce:2202: flush_pg_stats: seq=171798691900 2024-02-05T05:52:13.130 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691900 2024-02-05T05:52:13.130 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673067 2-64424509537 3-309237645340 4-107374182481 5-476741369859 6-150323855428 7-171798691900' 2024-02-05T05:52:13.130 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:13.130 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:52:13.130 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-02-05T05:52:13.130 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-02-05T05:52:13.130 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673067 2-64424509537 3-309237645340 4-107374182481 5-476741369859 6-150323855428 7-171798691900 8-193273528369' 2024-02-05T05:52:13.130 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:13.130 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:52:13.130 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467972 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467972 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673067 2-64424509537 3-309237645340 4-107374182481 5-476741369859 6-150323855428 7-171798691900 8-193273528369 9-463856467972' 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836587 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836587 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836587 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: waiting osd.0 seq 21474836587 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836587 -lt 21474836587 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:13.131 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:13.132 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673067 2024-02-05T05:52:13.132 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:52:13.132 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673067 2024-02-05T05:52:13.132 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:13.132 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673067 2024-02-05T05:52:13.132 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673067' 2024-02-05T05:52:13.132 INFO:tasks.workunit.client.0.smithi148.stderr:264596: waiting osd.1 seq 42949673067 2024-02-05T05:52:13.574 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221phtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509535 2024-02-05T05:52:13.575 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:13.575 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509535 2024-02-05T05:52:13.575 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509535' 2024-02-05T05:52:13.575 INFO:tasks.workunit.client.0.smithi148.stderr:264598: waiting osd.2 seq 64424509535 2024-02-05T05:52:13.575 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:52:13.575 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509535 2024-02-05T05:52:13.575 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:13.575 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645339 2024-02-05T05:52:13.575 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:13.575 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:52:13.575 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645339 2024-02-05T05:52:13.576 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:13.576 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645339 2024-02-05T05:52:13.576 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645339' 2024-02-05T05:52:13.576 INFO:tasks.workunit.client.0.smithi148.stderr:264598: waiting osd.3 seq 309237645339 2024-02-05T05:52:13.576 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:52:13.576 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645341 -lt 309237645339 2024-02-05T05:52:13.576 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:13.576 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182480 2024-02-05T05:52:13.576 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:13.576 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:52:13.576 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182480 2024-02-05T05:52:13.576 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:13.576 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182480 2024-02-05T05:52:13.576 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182480' 2024-02-05T05:52:13.577 INFO:tasks.workunit.client.0.smithi148.stderr:264598: waiting osd.4 seq 107374182480 2024-02-05T05:52:13.577 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:52:13.577 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182482 -lt 107374182480 2024-02-05T05:52:13.577 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:13.577 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-476741369858 2024-02-05T05:52:13.577 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:13.577 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:52:13.577 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-476741369858 2024-02-05T05:52:13.577 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:13.577 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369858 2024-02-05T05:52:14.487 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats4: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:52:14.488 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673067 -lt 42949673067 2024-02-05T05:52:14.488 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:14.488 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509537 2024-02-05T05:52:14.488 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:14.488 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:52:14.488 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509537 2024-02-05T05:52:14.488 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:14.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509537 2024-02-05T05:52:14.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-02-05T05:52:14.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: waiting osd.2 seq 64424509537 2024-02-05T05:52:14.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:52:14.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509537 2024-02-05T05:52:14.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:14.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645340 2024-02-05T05:52:14.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:14.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:52:14.491 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645340 2024-02-05T05:52:14.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:14.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645340 2024-02-05T05:52:14.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645340' 2024-02-05T05:52:14.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: waiting osd.3 seq 309237645340 2024-02-05T05:52:14.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:52:14.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645341 -lt 309237645340 2024-02-05T05:52:14.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:14.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182481 2024-02-05T05:52:14.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:14.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:52:14.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182481 2024-02-05T05:52:14.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:14.492 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182481 2024-02-05T05:52:14.493 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-02-05T05:52:14.493 INFO:tasks.workunit.client.0.smithi148.stderr:264596: waiting osd.4 seq 107374182481 2024-02-05T05:52:14.493 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:52:14.493 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182482 -lt 107374182481 2024-02-05T05:52:14.493 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:14.493 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-476741369859 2024-02-05T05:52:14.904 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //h: echo 'waiting osd.5 seq 476741369858' 2024-02-05T05:52:14.904 INFO:tasks.workunit.client.0.smithi148.stderr:264598: waiting osd.5 seq 476741369858 2024-02-05T05:52:14.904 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:52:14.904 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369859 -lt 476741369858 2024-02-05T05:52:14.904 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:14.904 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855426 2024-02-05T05:52:14.904 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:14.904 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:52:14.905 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855426 2024-02-05T05:52:14.905 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:14.905 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855426 2024-02-05T05:52:14.905 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855426' 2024-02-05T05:52:14.905 INFO:tasks.workunit.client.0.smithi148.stderr:264598: waiting osd.6 seq 150323855426 2024-02-05T05:52:14.905 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:52:14.905 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855428 -lt 150323855426 2024-02-05T05:52:14.905 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:14.905 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691899 2024-02-05T05:52:14.905 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:14.905 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:52:14.905 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691899 2024-02-05T05:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691899 2024-02-05T05:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691899' 2024-02-05T05:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:264598: waiting osd.7 seq 171798691899 2024-02-05T05:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691901 -lt 171798691899 2024-02-05T05:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528367 2024-02-05T05:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528367 2024-02-05T05:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528367 2024-02-05T05:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528367' 2024-02-05T05:52:14.907 INFO:tasks.workunit.client.0.smithi148.stderr:264598: waiting osd.8 seq 193273528367 2024-02-05T05:52:14.907 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:52:14.907 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528369 -lt 193273528367 2024-02-05T05:52:15.430 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:ome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:15.431 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:52:15.431 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-476741369859 2024-02-05T05:52:15.431 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:15.431 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369859 2024-02-05T05:52:15.431 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 476741369859' 2024-02-05T05:52:15.431 INFO:tasks.workunit.client.0.smithi148.stderr:264596: waiting osd.5 seq 476741369859 2024-02-05T05:52:15.431 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:52:15.431 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369860 -lt 476741369859 2024-02-05T05:52:15.431 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:15.431 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855428 2024-02-05T05:52:15.431 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:15.431 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:52:15.431 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855428 2024-02-05T05:52:15.431 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:15.432 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855428 2024-02-05T05:52:15.432 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855428' 2024-02-05T05:52:15.432 INFO:tasks.workunit.client.0.smithi148.stderr:264596: waiting osd.6 seq 150323855428 2024-02-05T05:52:15.432 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:52:15.432 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855428 -lt 150323855428 2024-02-05T05:52:15.432 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:15.432 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:15.432 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691900 2024-02-05T05:52:15.432 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:52:15.432 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691900 2024-02-05T05:52:15.433 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:15.433 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691900 2024-02-05T05:52:15.433 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-02-05T05:52:15.433 INFO:tasks.workunit.client.0.smithi148.stderr:264596: waiting osd.7 seq 171798691900 2024-02-05T05:52:15.433 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:52:15.433 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691901 -lt 171798691900 2024-02-05T05:52:15.433 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:15.433 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-02-05T05:52:15.433 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:15.433 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:52:15.433 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-02-05T05:52:16.254 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:16.254 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-463856467971 2024-02-05T05:52:16.254 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:16.254 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:52:16.254 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-463856467971 2024-02-05T05:52:16.254 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:16.254 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467971 2024-02-05T05:52:16.254 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 463856467971' 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: waiting osd.9 seq 463856467971 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467972 -lt 463856467971 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:52:16.255 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:52:16.256 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:52:16.256 INFO:tasks.workunit.client.0.smithi148.stderr:264598: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:52:16.256 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:52:16.256 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:52:16.256 INFO:tasks.workunit.client.0.smithi148.stderr:264598: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:52:16.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:52:17.216 INFO:tasks.workunit.client.0.smithi148.stderr:_pg_stats: cut -d - -f 2 2024-02-05T05:52:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-02-05T05:52:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-02-05T05:52:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:264596: waiting osd.8 seq 193273528369 2024-02-05T05:52:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:52:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528369 -lt 193273528369 2024-02-05T05:52:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-463856467972 2024-02-05T05:52:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:52:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-463856467972 2024-02-05T05:52:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467972 2024-02-05T05:52:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 463856467972' 2024-02-05T05:52:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:264596: waiting osd.9 seq 463856467972 2024-02-05T05:52:17.218 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:52:17.218 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467972 -lt 463856467972 2024-02-05T05:52:17.218 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:52:17.218 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:52:17.218 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:52:17.218 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:52:17.218 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:52:17.218 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:52:17.218 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:52:17.218 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:52:17.218 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:52:17.219 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:52:17.219 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:52:17.219 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:52:17.219 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:52:17.219 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:52:17.219 INFO:tasks.workunit.client.0.smithi148.stderr:264596: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:52:17.219 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:52:17.220 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:52:17.220 INFO:tasks.workunit.client.0.smithi148.stderr:264596: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:52:17.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:52:17.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:52:17.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 264594 2024-02-05T05:52:17.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:52:17.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:52:17.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:52:17.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:52:17.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:52:17.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:52:17.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:52:17.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-02-05T05:52:17.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:52:17.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:52:17.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-02-05T05:52:17.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-02-05T05:52:17.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:52:17.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:52:17.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:52:17.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:52:17.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 267850"' 2024-02-05T05:52:17.303 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 267850' 2024-02-05T05:52:17.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:52:17.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:52:17.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:52:17.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 267852"' 2024-02-05T05:52:17.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:52:17.304 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 267852' 2024-02-05T05:52:17.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:52:17.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 267850 267852' 2024-02-05T05:52:17.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/267853: /' 2024-02-05T05:52:17.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:52:17.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:52:17.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 267850 2024-02-05T05:52:17.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:52:17.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/267855: /' 2024-02-05T05:52:21.338 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:52:21.338 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:52:21.338 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:52:21.338 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:52:21.339 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:52:21.339 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:52:21.339 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:52:21.339 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:52:21.339 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:52:21.339 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:52:21.339 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:52:21.339 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:52:21.339 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:52:21.339 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:52:21.339 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:52:21.339 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:52:21.339 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:52:21.339 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:52:21.340 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:52:21.340 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:52:21.340 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:52:21.340 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:52:21.340 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:52:21.340 INFO:tasks.workunit.client.0.smithi148.stderr:267853: remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:52:21.340 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:52:21.340 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:52:21.340 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:52:21.340 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:52:21.340 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:52:21.340 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:52:21.340 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:52:21.341 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:52:21.343 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:52:21.343 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:52:21.343 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:52:21.343 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:52:21.343 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:52:21.343 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:52:21.343 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:52:21.344 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:52:21.344 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:52:21.344 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:52:21.344 INFO:tasks.workunit.client.0.smithi148.stderr:267853: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:52:21.344 INFO:tasks.workunit.client.0.smithi148.stderr:267853: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:21.344 INFO:tasks.workunit.client.0.smithi148.stderr:267853: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:52:21.344 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:21.344 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:21.344 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:52:21.344 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:52:21.344 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:52:21.344 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:52:21.344 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:52:21.344 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:52:21.345 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:52:21.345 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:52:21.345 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:52:21.345 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:52:21.345 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:52:21.345 INFO:tasks.workunit.client.0.smithi148.stderr:267853: start osd.3 2024-02-05T05:52:21.345 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:52:21.376 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:52:21.376 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:52:21.376 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:52:21.376 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:52:21.376 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:52:21.376 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:52:21.376 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:52:21.377 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:52:21.378 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-02-05T05:52:21.378 INFO:tasks.workunit.client.0.smithi148.stderr:267855: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:52:21.378 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:52:21.378 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:52:21.378 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:52:21.378 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:52:21.378 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:52:21.378 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:52:21.378 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:52:21.378 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:52:21.380 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:52:21.380 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:52:21.380 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:52:21.380 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:52:21.380 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:52:21.381 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:52:21.381 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:52:21.381 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:52:21.381 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:52:21.381 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:52:21.381 INFO:tasks.workunit.client.0.smithi148.stderr:267855: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:52:21.381 INFO:tasks.workunit.client.0.smithi148.stderr:267855: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:21.381 INFO:tasks.workunit.client.0.smithi148.stderr:267855: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:52:21.381 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:21.381 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:21.381 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:52:21.381 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:52:21.381 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:52:21.381 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:52:21.382 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:52:21.382 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:52:21.382 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:52:21.382 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:52:21.382 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:52:21.382 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:52:21.382 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:52:21.382 INFO:tasks.workunit.client.0.smithi148.stderr:267855: start osd.5 2024-02-05T05:52:21.382 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:52:26.088 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:52:26.088 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:52:26.089 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:52:26.089 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:52:26.089 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:52:26.089 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 2024-02-05T05:52:21.393+0000 7f1927065640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:26.089 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 2024-02-05T05:52:21.397+0000 7f1927065640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:26.089 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 2024-02-05T05:52:21.397+0000 7f1927065640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:26.089 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:52:26.089 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:52:26.089 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:52:26.089 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:52:26.089 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:52:26.089 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:26.089 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:52:26.090 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 0 2024-02-05T05:52:26.090 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:26.090 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:52:26.090 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 2024-02-05T05:52:21.937+0000 7f1927065640 -1 Falling back to public interface 2024-02-05T05:52:26.090 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:26.090 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:26.090 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:26.090 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:52:26.090 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 1 2024-02-05T05:52:26.090 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:26.090 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:52:26.090 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:26.090 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:26.091 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:26.091 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:52:26.091 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 2 2024-02-05T05:52:26.091 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:26.091 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:52:26.091 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 2024-02-05T05:52:24.681+0000 7f1927065640 -1 osd.3 113 log_to_monitors true 2024-02-05T05:52:26.091 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:26.091 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:26.091 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:26.091 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:52:26.091 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 3 2024-02-05T05:52:26.091 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:26.091 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:52:26.108 INFO:tasks.workunit.client.0.smithi148.stderr:267853: osd.3 up in weight 1 up_from 118 up_thru 118 down_at 114 l267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:52:26.108 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:52:26.108 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:52:26.108 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:52:26.108 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:52:26.108 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 2024-02-05T05:52:21.429+0000 7f7f35e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:26.108 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 2024-02-05T05:52:21.433+0000 7f7f35e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:26.108 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 2024-02-05T05:52:21.437+0000 7f7f35e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:26.108 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:52:26.108 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:52:26.109 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:52:26.109 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:52:26.109 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:52:26.109 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:26.109 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:52:26.109 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 0 2024-02-05T05:52:26.109 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:26.109 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:52:26.109 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 2024-02-05T05:52:21.985+0000 7f7f35e0f640 -1 Falling back to public interface 2024-02-05T05:52:26.109 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:26.109 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:26.109 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:26.109 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 1 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 2 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 2024-02-05T05:52:25.217+0000 7f7f35e0f640 -1 osd.5 113 log_to_monitors true 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:26.110 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:52:26.111 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 3 2024-02-05T05:52:26.111 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:26.111 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:52:27.388 INFO:tasks.workunit.client.0.smithi148.stderr:267855: osd.5 up in weight 1 up_from 119 up_thru 111 down_at 114 last_clean_interval [72,113) [v2:127.0.0.1:6826/1054381624,v1:127.0.0.1:6827/1054381624] [v2:127.0.0.1:6828/1054381624,v1:127.0.0.1:6829/1054381624] exists,up 5498ffc6-6c90-4948-b650-4948ddc04afa 2024-02-05T05:52:27.388 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:52:27.388 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:52:27.388 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:52:27.389 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:52:27.389 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:52:27.389 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:52:27.389 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:52:27.389 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:52:27.389 INFO:tasks.workunit.client.0.smithi148.stderr:267853: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:52:27.389 INFO:tasks.workunit.client.0.smithi148.stderr:267853: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:52:27.389 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:52:27.389 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:52:27.389 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:52:27.389 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:52:27.389 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:52:27.390 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:52:27.390 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:52:27.390 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:52:27.390 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:52:27.390 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:52:27.390 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 1 2024-02-05T05:52:27.390 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 2 2024-02-05T05:52:27.390 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 3 2024-02-05T05:52:27.390 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 4 2024-02-05T05:52:27.390 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 5 2024-02-05T05:52:27.390 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 6 2024-02-05T05:52:27.390 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 7 2024-02-05T05:52:27.390 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 8 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267853: 9' 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836591 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836591 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591' 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673072 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673072 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673072' 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.clienast_clean_interval [111,113) [v2:127.0.0.1:6874/1361883522,v1:127.0.0.1:6875/1361883522] [v2:127.0.0.1:6876/1361883522,v1:127.0.0.1:6877/1361883522] exists,up 9832157c-d831-4b53-9b05-71ea506d914b 2024-02-05T05:52:27.391 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:52:27.392 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:52:27.392 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:52:27.392 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:52:27.392 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:52:27.392 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:52:27.392 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:52:27.392 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:52:27.392 INFO:tasks.workunit.client.0.smithi148.stderr:267855: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:52:27.392 INFO:tasks.workunit.client.0.smithi148.stderr:267855: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:52:27.392 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:52:27.392 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:52:27.392 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:52:27.392 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:52:27.393 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:52:27.393 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:52:27.393 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:52:27.393 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:52:27.393 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:52:27.393 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:52:27.393 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 1 2024-02-05T05:52:27.393 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 2 2024-02-05T05:52:27.393 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 3 2024-02-05T05:52:27.393 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 4 2024-02-05T05:52:27.393 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 5 2024-02-05T05:52:27.393 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 6 2024-02-05T05:52:27.393 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 7 2024-02-05T05:52:27.394 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 8 2024-02-05T05:52:27.394 INFO:tasks.workunit.client.0.smithi148.stderr:267855: 9' 2024-02-05T05:52:27.394 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:52:27.394 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:27.394 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:52:27.394 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836592 2024-02-05T05:52:27.394 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836592 2024-02-05T05:52:27.394 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592' 2024-02-05T05:52:27.394 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:27.394 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:52:27.394 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673071 2024-02-05T05:52:27.394 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673071 2024-02-05T05:52:27.394 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673071' 2024-02-05T05:52:27.394 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:28.871 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.cliet.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:52:28.872 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509541 2024-02-05T05:52:28.872 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509541 2024-02-05T05:52:28.872 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673072 2-64424509541' 2024-02-05T05:52:28.872 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:28.872 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:52:28.872 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140931 2024-02-05T05:52:28.872 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140931 2024-02-05T05:52:28.872 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673072 2-64424509541 3-506806140931' 2024-02-05T05:52:28.872 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:28.873 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:52:28.873 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182486 2024-02-05T05:52:28.873 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182486 2024-02-05T05:52:28.873 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673072 2-64424509541 3-506806140931 4-107374182486' 2024-02-05T05:52:28.873 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:28.873 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:52:28.873 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=511101108226 2024-02-05T05:52:28.873 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 511101108226 2024-02-05T05:52:28.873 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673072 2-64424509541 3-506806140931 4-107374182486 5-511101108226' 2024-02-05T05:52:28.873 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:28.873 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:52:28.873 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855433 2024-02-05T05:52:28.874 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855433 2024-02-05T05:52:28.874 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673072 2-64424509541 3-506806140931 4-107374182486 5-511101108226 6-150323855433' 2024-02-05T05:52:28.874 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:28.874 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:52:28.874 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691904 2024-02-05T05:52:28.874 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691904 2024-02-05T05:52:28.874 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673072 2-64424509541 3-506806140931 4-107374182486 5-511101108226 6-150323855433 7-171798691904' 2024-02-05T05:52:28.874 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:28.874 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:52:28.878 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtestnt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:52:28.878 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509542 2024-02-05T05:52:28.878 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509542 2024-02-05T05:52:28.878 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673071 2-64424509542' 2024-02-05T05:52:28.878 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:28.878 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:52:28.878 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140930 2024-02-05T05:52:28.878 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140930 2024-02-05T05:52:28.878 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673071 2-64424509542 3-506806140930' 2024-02-05T05:52:28.878 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:28.878 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:52:28.878 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182487 2024-02-05T05:52:28.879 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182487 2024-02-05T05:52:28.879 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673071 2-64424509542 3-506806140930 4-107374182487' 2024-02-05T05:52:28.879 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:28.879 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:52:28.879 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=511101108227 2024-02-05T05:52:28.879 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 511101108227 2024-02-05T05:52:28.879 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673071 2-64424509542 3-506806140930 4-107374182487 5-511101108227' 2024-02-05T05:52:28.879 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:28.879 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:52:28.879 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855432 2024-02-05T05:52:28.879 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855432 2024-02-05T05:52:28.879 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673071 2-64424509542 3-506806140930 4-107374182487 5-511101108227 6-150323855432' 2024-02-05T05:52:28.879 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:28.880 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:52:28.880 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691905 2024-02-05T05:52:28.880 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691905 2024-02-05T05:52:28.880 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673071 2-64424509542 3-506806140930 4-107374182487 5-511101108227 6-150323855432 7-171798691905' 2024-02-05T05:52:28.880 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:28.880 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:52:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528374 2024-02-05T05:52:29.684 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528374 2024-02-05T05:52:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673071 2-64424509542 3-506806140930 4-107374182487 5-511101108227 6-150323855432 7-171798691905 8-193273528374' 2024-02-05T05:52:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:52:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467977 2024-02-05T05:52:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467977 2024-02-05T05:52:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673071 2-64424509542 3-506806140930 4-107374182487 5-511101108227 6-150323855432 7-171798691905 8-193273528374 9-463856467977' 2024-02-05T05:52:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836592 2024-02-05T05:52:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:52:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836592 2024-02-05T05:52:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:29.685 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836592 2024-02-05T05:52:29.686 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2024-02-05T05:52:29.686 INFO:tasks.workunit.client.0.smithi148.stderr:267855: waiting osd.0 seq 21474836592 2024-02-05T05:52:29.686 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:52:29.686 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836593 -lt 21474836592 2024-02-05T05:52:29.686 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:29.686 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673071 2024-02-05T05:52:29.686 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:29.686 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:52:29.686 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673071 2024-02-05T05:52:29.686 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:29.686 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673071 2024-02-05T05:52:29.686 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673071' 2024-02-05T05:52:29.686 INFO:tasks.workunit.client.0.smithi148.stderr:267855: waiting osd.1 seq 42949673071 2024-02-05T05:52:29.686 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:52:29.687 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673072 -lt 42949673071 2024-02-05T05:52:29.687 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:29.687 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509542 2024-02-05T05:52:29.687 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:29.687 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:52:29.689 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest//clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528373 2024-02-05T05:52:29.690 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528373 2024-02-05T05:52:29.690 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673072 2-64424509541 3-506806140931 4-107374182486 5-511101108226 6-150323855433 7-171798691904 8-193273528373' 2024-02-05T05:52:29.690 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:29.690 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:52:29.690 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467976 2024-02-05T05:52:29.690 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467976 2024-02-05T05:52:29.690 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673072 2-64424509541 3-506806140931 4-107374182486 5-511101108226 6-150323855433 7-171798691904 8-193273528373 9-463856467976' 2024-02-05T05:52:29.690 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:29.690 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836591 2024-02-05T05:52:29.690 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:29.690 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:52:29.690 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836591 2024-02-05T05:52:29.691 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:29.691 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836591 2024-02-05T05:52:29.691 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-02-05T05:52:29.691 INFO:tasks.workunit.client.0.smithi148.stderr:267853: waiting osd.0 seq 21474836591 2024-02-05T05:52:29.691 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:52:29.691 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836593 -lt 21474836591 2024-02-05T05:52:29.691 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:29.691 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673072 2024-02-05T05:52:29.691 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:29.691 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:52:29.691 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673072 2024-02-05T05:52:29.691 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:29.691 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673072 2024-02-05T05:52:29.691 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673072' 2024-02-05T05:52:29.692 INFO:tasks.workunit.client.0.smithi148.stderr:267853: waiting osd.1 seq 42949673072 2024-02-05T05:52:29.692 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:52:29.692 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673072 -lt 42949673072 2024-02-05T05:52:29.692 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:29.692 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509541 2024-02-05T05:52:29.692 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:29.692 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:52:30.720 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509541 2024-02-05T05:52:30.720 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:30.720 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509541 2024-02-05T05:52:30.720 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509541' 2024-02-05T05:52:30.720 INFO:tasks.workunit.client.0.smithi148.stderr:267853: waiting osd.2 seq 64424509541 2024-02-05T05:52:30.720 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:52:30.720 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509542 -lt 64424509541 2024-02-05T05:52:30.720 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:30.720 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-506806140931 2024-02-05T05:52:30.720 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:30.721 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:52:30.721 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-506806140931 2024-02-05T05:52:30.721 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:30.721 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140931 2024-02-05T05:52:30.721 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 506806140931' 2024-02-05T05:52:30.721 INFO:tasks.workunit.client.0.smithi148.stderr:267853: waiting osd.3 seq 506806140931 2024-02-05T05:52:30.721 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:52:30.721 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140931 -lt 506806140931 2024-02-05T05:52:30.721 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:30.721 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182486 2024-02-05T05:52:30.721 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:30.721 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:52:30.721 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182486 2024-02-05T05:52:30.721 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:30.722 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182486 2024-02-05T05:52:30.722 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182486' 2024-02-05T05:52:30.722 INFO:tasks.workunit.client.0.smithi148.stderr:267853: waiting osd.4 seq 107374182486 2024-02-05T05:52:30.722 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:52:30.722 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182487 -lt 107374182486 2024-02-05T05:52:30.722 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:30.722 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-511101108226 2024-02-05T05:52:30.722 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:30.722 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:52:30.722 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-511101108226 2024-02-05T05:52:30.722 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:30.722 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=511101108226 2024-02-05T05:52:30.722 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509542 2024-02-05T05:52:30.722 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509542 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509542' 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: waiting osd.2 seq 64424509542 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509542 -lt 64424509542 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-506806140930 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-506806140930 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140930 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 506806140930' 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: waiting osd.3 seq 506806140930 2024-02-05T05:52:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:52:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140931 -lt 506806140930 2024-02-05T05:52:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182487 2024-02-05T05:52:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:52:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182487 2024-02-05T05:52:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182487 2024-02-05T05:52:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182487' 2024-02-05T05:52:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:267855: waiting osd.4 seq 107374182487 2024-02-05T05:52:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:52:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182487 -lt 107374182487 2024-02-05T05:52:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:30.725 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-511101108227 2024-02-05T05:52:30.725 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:30.725 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:52:30.725 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-511101108227 2024-02-05T05:52:30.725 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:30.725 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=511101108227 2024-02-05T05:52:32.023 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo'waiting osd.5 seq 511101108226' 2024-02-05T05:52:32.023 INFO:tasks.workunit.client.0.smithi148.stderr:267853: waiting osd.5 seq 511101108226 2024-02-05T05:52:32.023 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:52:32.023 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 511101108227 -lt 511101108226 2024-02-05T05:52:32.023 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:32.023 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855433 2024-02-05T05:52:32.024 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:32.024 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:52:32.024 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855433 2024-02-05T05:52:32.024 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:32.024 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855433 2024-02-05T05:52:32.024 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855433' 2024-02-05T05:52:32.024 INFO:tasks.workunit.client.0.smithi148.stderr:267853: waiting osd.6 seq 150323855433 2024-02-05T05:52:32.024 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:52:32.024 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855433 -lt 150323855433 2024-02-05T05:52:32.024 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:32.024 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691904 2024-02-05T05:52:32.024 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:32.024 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691904 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691904 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691904' 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: waiting osd.7 seq 171798691904 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691906 -lt 171798691904 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528373 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528373 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528373 2024-02-05T05:52:32.025 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528373' 2024-02-05T05:52:32.026 INFO:tasks.workunit.client.0.smithi148.stderr:267853: waiting osd.8 seq 193273528373 2024-02-05T05:52:32.026 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:52:32.026 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528374 -lt 193273528373 2024-02-05T05:52:32.030 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fl 'waiting osd.5 seq 511101108227' 2024-02-05T05:52:32.030 INFO:tasks.workunit.client.0.smithi148.stderr:267855: waiting osd.5 seq 511101108227 2024-02-05T05:52:32.030 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:52:32.030 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 511101108227 -lt 511101108227 2024-02-05T05:52:32.030 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:32.031 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855432 2024-02-05T05:52:32.031 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:32.031 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:52:32.031 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855432 2024-02-05T05:52:32.031 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:32.031 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855432 2024-02-05T05:52:32.031 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855432' 2024-02-05T05:52:32.031 INFO:tasks.workunit.client.0.smithi148.stderr:267855: waiting osd.6 seq 150323855432 2024-02-05T05:52:32.031 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:52:32.031 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855433 -lt 150323855432 2024-02-05T05:52:32.031 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:32.031 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691905 2024-02-05T05:52:32.031 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:52:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691905 2024-02-05T05:52:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691905 2024-02-05T05:52:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691905' 2024-02-05T05:52:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:267855: waiting osd.7 seq 171798691905 2024-02-05T05:52:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:52:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691906 -lt 171798691905 2024-02-05T05:52:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528374 2024-02-05T05:52:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:52:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528374 2024-02-05T05:52:32.032 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:32.033 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528374 2024-02-05T05:52:32.033 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528374' 2024-02-05T05:52:32.033 INFO:tasks.workunit.client.0.smithi148.stderr:267855: waiting osd.8 seq 193273528374 2024-02-05T05:52:32.033 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:52:32.033 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528374 -lt 193273528374 2024-02-05T05:52:33.422 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:33.422 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-463856467977 2024-02-05T05:52:33.422 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:33.422 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:52:33.423 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-463856467977 2024-02-05T05:52:33.423 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:33.423 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467977 2024-02-05T05:52:33.423 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 463856467977' 2024-02-05T05:52:33.423 INFO:tasks.workunit.client.0.smithi148.stderr:267855: waiting osd.9 seq 463856467977 2024-02-05T05:52:33.423 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:52:33.423 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467977 -lt 463856467977 2024-02-05T05:52:33.423 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:52:33.423 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:52:33.423 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:52:33.423 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:52:33.423 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:52:33.424 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:52:33.424 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:52:33.424 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:52:33.424 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:52:33.424 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:52:33.424 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:52:33.424 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:52:33.424 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:52:33.424 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:52:33.424 INFO:tasks.workunit.client.0.smithi148.stderr:267855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:52:33.424 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:52:33.425 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:52:33.425 INFO:tasks.workunit.client.0.smithi148.stderr:267855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:52:33.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:52:33.453 INFO:tasks.workunit.client.0.smithi148.stderr:ush_pg_stats: for s in $seqs 2024-02-05T05:52:33.453 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-463856467976 2024-02-05T05:52:33.453 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:33.453 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:52:33.454 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-463856467976 2024-02-05T05:52:33.454 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:33.454 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467976 2024-02-05T05:52:33.454 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 463856467976' 2024-02-05T05:52:33.454 INFO:tasks.workunit.client.0.smithi148.stderr:267853: waiting osd.9 seq 463856467976 2024-02-05T05:52:33.454 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:52:33.454 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467977 -lt 463856467976 2024-02-05T05:52:33.454 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:52:33.454 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:52:33.454 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:52:33.454 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:52:33.454 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:52:33.454 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:52:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:52:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:52:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:52:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:52:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:52:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:52:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:52:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:52:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:267853: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:52:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:52:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:52:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:267853: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:52:33.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:52:33.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:52:33.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 267852 2024-02-05T05:52:33.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:52:33.456 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:52:33.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:52:33.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:52:33.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:52:33.456 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:52:33.456 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:52:33.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:52:33.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:52:33.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:52:33.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:52:33.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:52:33.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:52:33.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:52:33.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:52:33.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:52:33.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:52:33.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:52:34.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:51:54.839405+0000 2024-02-05T05:52:34.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:52:34.348 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:52:34.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:51:54.839405+0000 2024-02-05T05:52:34.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:52:34.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:51:54.839405+0000 2024-02-05T05:52:34.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:52:34.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:52:34.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:52:34.366 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:52:34.366 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:52:34.366 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:52:34.366 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:52:34.366 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:52:34.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:51:54.839405+0000 '>' 2024-02-05T05:51:54.839405+0000 2024-02-05T05:52:34.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:52:35.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:52:35.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:52:35.656 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:52:35.656 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:52:35.656 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:52:35.657 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:52:35.657 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:52:36.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:51:54.839405+0000 '>' 2024-02-05T05:51:54.839405+0000 2024-02-05T05:52:36.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:52:37.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:52:37.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:52:37.029 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:52:37.029 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:52:37.029 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:52:37.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:52:37.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:52:37.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:52:34.758482+0000 '>' 2024-02-05T05:51:54.839405+0000 2024-02-05T05:52:37.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:52:37.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:52:37.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:52:37.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:52:37.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:52:37.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 271208"' 2024-02-05T05:52:37.380 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 271208' 2024-02-05T05:52:37.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:52:37.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:52:37.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:52:37.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 271209"' 2024-02-05T05:52:37.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 271209' 2024-02-05T05:52:37.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:52:37.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 271208 271209' 2024-02-05T05:52:37.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:52:37.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:52:37.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 271208 2024-02-05T05:52:37.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/271211: /' 2024-02-05T05:52:37.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:52:37.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:52:37.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/271213: /' 2024-02-05T05:52:40.718 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:52:40.718 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:52:40.718 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:52:40.719 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:52:40.719 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:52:40.719 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:52:40.719 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:52:40.719 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:52:40.719 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:52:40.719 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:52:40.719 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:52:40.719 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:52:40.719 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:52:40.719 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:52:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:52:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:52:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:52:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:52:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:52:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:52:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:52:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:52:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:52:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:271211: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:52:40.720 INFO:tasks.workunit.client.0.smithi148.stderr:271211: _ 2024-02-05T05:52:40.721 INFO:tasks.workunit.client.0.smithi148.stderr:271211: hinfo_key 2024-02-05T05:52:40.721 INFO:tasks.workunit.client.0.smithi148.stderr:271211: snapset 2024-02-05T05:52:40.721 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:52:40.721 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:52:40.721 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:52:40.721 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:52:40.721 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:52:40.721 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:52:40.721 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:52:40.723 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:52:40.723 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:52:40.723 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:52:40.723 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:52:40.723 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:52:40.723 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:52:40.723 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:52:40.723 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:52:40.723 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:52:40.723 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:52:40.724 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:52:40.724 INFO:tasks.workunit.client.0.smithi148.stderr:271211: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:52:40.724 INFO:tasks.workunit.client.0.smithi148.stderr:271211: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:40.724 INFO:tasks.workunit.client.0.smithi148.stderr:271211: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:52:40.724 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:40.724 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:40.724 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:52:40.724 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:52:40.724 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:52:40.724 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:52:40.724 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:52:40.725 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:52:40.725 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:52:40.725 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:52:40.725 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:52:40.725 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:52:40.725 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:52:40.725 INFO:tasks.workunit.client.0.smithi148.stderr:271211: start osd.3 2024-02-05T05:52:41.140 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:52:41.140 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:52:41.140 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:52:41.140 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:52:41.140 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:52:41.140 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:52:41.140 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:52:41.140 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:52:41.140 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:52:41.140 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:52:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:52:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:52:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:52:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:52:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:52:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:52:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:52:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:52:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:52:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:52:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:52:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:52:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-02-05T05:52:41.142 INFO:tasks.workunit.client.0.smithi148.stderr:271213: _ 2024-02-05T05:52:41.142 INFO:tasks.workunit.client.0.smithi148.stderr:271213: hinfo_key 2024-02-05T05:52:41.142 INFO:tasks.workunit.client.0.smithi148.stderr:271213: snapset 2024-02-05T05:52:41.142 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:52:41.142 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:52:41.142 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:52:41.142 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:52:41.142 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:52:41.142 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:52:41.142 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:52:41.142 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:52:41.144 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:52:41.144 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:52:41.144 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:52:41.144 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:52:41.144 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:52:41.144 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:52:41.144 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:52:41.144 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:52:41.144 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:52:41.144 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:52:41.145 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:52:41.146 INFO:tasks.workunit.client.0.smithi148.stderr:271213: start osd.5 2024-02-05T05:52:45.020 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:52:45.020 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:52:45.020 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:52:45.020 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:52:45.020 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:52:45.020 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:52:45.020 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 2024-02-05T05:52:40.772+0000 7f52d4e47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 2024-02-05T05:52:40.784+0000 7f52d4e47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 2024-02-05T05:52:40.788+0000 7f52d4e47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 0 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 2024-02-05T05:52:41.332+0000 7f52d4e47640 -1 Falling back to public interface 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:52:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 1 2024-02-05T05:52:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:52:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:52:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 2 2024-02-05T05:52:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:52:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 2024-02-05T05:52:44.648+0000 7f52d4e47640 -1 osd.3 121 log_to_monitors true 2024-02-05T05:52:45.023 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:45.023 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:45.023 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:52:45.023 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 3 2024-02-05T05:52:45.023 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:46.262 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-02-05T05:52:46.262 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:52:46.262 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:52:46.263 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:52:46.263 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:52:46.263 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:52:46.263 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 2024-02-05T05:52:41.192+0000 7f16a73cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:46.263 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 2024-02-05T05:52:41.200+0000 7f16a73cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:46.263 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 2024-02-05T05:52:41.204+0000 7f16a73cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:46.263 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:52:46.263 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:52:46.263 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:52:46.263 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:52:46.263 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:52:46.263 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:46.264 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:52:46.264 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 0 2024-02-05T05:52:46.264 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:46.264 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:52:46.264 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 2024-02-05T05:52:41.756+0000 7f16a73cb640 -1 Falling back to public interface 2024-02-05T05:52:46.264 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:46.264 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:46.264 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:46.264 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:52:46.264 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 1 2024-02-05T05:52:46.264 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:46.264 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 2 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 2024-02-05T05:52:44.500+0000 7f16a73cb640 -1 osd.5 121 log_to_monitors true 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 3 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:46.265 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:52:47.359 INFO:tasks.workunit.client.0.smithi148.stderr:271213: osd.5 up in weight 1 up_from 127 up_thru 119 down_at 122 last_clean_interval [119,121) [v2:127.0.0.1:6874/3938753643,v1:127.0.0.1:6875/3938753643] [v2:127.0.0.1:6876/3938753643,v1:127.0.0.1:6877/3938753643] exists,up 9832157c-d831-4b53-9b05-71ea506d914b 2024-02-05T05:52:47.359 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:52:47.359 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:52:47.359 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:52:47.359 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:52:47.359 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:52:47.359 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:52:47.359 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:52:47.360 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:52:47.360 INFO:tasks.workunit.client.0.smithi148.stderr:271213: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:52:47.360 INFO:tasks.workunit.client.0.smithi148.stderr:271213: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:52:47.360 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:52:47.360 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:52:47.360 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:52:47.360 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:52:47.360 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:52:47.360 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:52:47.360 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:52:47.360 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:52:47.360 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:52:47.360 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:52:47.361 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 1 2024-02-05T05:52:47.361 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 2 2024-02-05T05:52:47.361 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 3 2024-02-05T05:52:47.361 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 4 2024-02-05T05:52:47.361 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 5 2024-02-05T05:52:47.361 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 6 2024-02-05T05:52:47.361 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 7 2024-02-05T05:52:47.361 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 8 2024-02-05T05:52:47.361 INFO:tasks.workunit.client.0.smithi148.stderr:271213: 9' 2024-02-05T05:52:47.361 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:52:47.361 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:47.361 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:52:47.361 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836597 2024-02-05T05:52:47.362 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836597 2024-02-05T05:52:47.362 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597' 2024-02-05T05:52:47.362 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:47.362 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:52:47.362 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673077 2024-02-05T05:52:47.362 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673077 2024-02-05T05:52:47.362 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673077' 2024-02-05T05:52:47.362 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:47.499 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/rs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:52:47.499 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:47.499 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:47.499 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:47.500 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:52:47.500 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 4 2024-02-05T05:52:47.500 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:47.500 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:52:47.500 INFO:tasks.workunit.client.0.smithi148.stderr:271211: osd.3 up in weight 1 up_from 127 up_thru 127 down_at 122 last_clean_interval [118,121) [v2:127.0.0.1:6826/298132116,v1:127.0.0.1:6827/298132116] [v2:127.0.0.1:6828/298132116,v1:127.0.0.1:6829/298132116] exists,up 5498ffc6-6c90-4948-b650-4948ddc04afa 2024-02-05T05:52:47.500 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:52:47.500 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:52:47.500 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:52:47.500 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:52:47.500 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:52:47.500 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:52:47.500 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:52:47.501 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:52:47.501 INFO:tasks.workunit.client.0.smithi148.stderr:271211: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:52:47.501 INFO:tasks.workunit.client.0.smithi148.stderr:271211: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:52:47.501 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:52:47.501 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:52:47.501 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:52:47.501 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:52:47.501 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:52:47.501 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:52:47.501 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:52:47.501 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:52:47.501 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:52:47.501 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:52:47.501 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 1 2024-02-05T05:52:47.502 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 2 2024-02-05T05:52:47.502 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 3 2024-02-05T05:52:47.502 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 4 2024-02-05T05:52:47.502 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 5 2024-02-05T05:52:47.502 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 6 2024-02-05T05:52:47.502 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 7 2024-02-05T05:52:47.502 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 8 2024-02-05T05:52:47.502 INFO:tasks.workunit.client.0.smithi148.stderr:271211: 9' 2024-02-05T05:52:47.502 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:52:47.502 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:47.502 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:52:47.502 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836598 2024-02-05T05:52:47.502 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836598 2024-02-05T05:52:47.502 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598' 2024-02-05T05:52:48.538 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:52:48.538 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509547 2024-02-05T05:52:48.538 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509547 2024-02-05T05:52:48.538 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673077 2-64424509547' 2024-02-05T05:52:48.539 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:48.539 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:52:48.539 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846594 2024-02-05T05:52:48.539 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846594 2024-02-05T05:52:48.539 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673077 2-64424509547 3-545460846594' 2024-02-05T05:52:48.539 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:48.539 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:52:48.539 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182492 2024-02-05T05:52:48.539 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182492 2024-02-05T05:52:48.539 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673077 2-64424509547 3-545460846594 4-107374182492' 2024-02-05T05:52:48.539 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:48.539 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:52:48.539 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846594 2024-02-05T05:52:48.540 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846594 2024-02-05T05:52:48.540 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673077 2-64424509547 3-545460846594 4-107374182492 5-545460846594' 2024-02-05T05:52:48.540 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:48.540 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:52:48.540 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855438 2024-02-05T05:52:48.540 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855438 2024-02-05T05:52:48.540 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673077 2-64424509547 3-545460846594 4-107374182492 5-545460846594 6-150323855438' 2024-02-05T05:52:48.540 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:48.540 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:52:48.540 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691910 2024-02-05T05:52:48.540 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691910 2024-02-05T05:52:48.540 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673077 2-64424509547 3-545460846594 4-107374182492 5-545460846594 6-150323855438 7-171798691910' 2024-02-05T05:52:48.540 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:48.540 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:52:48.541 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubunhome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:48.541 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:52:48.541 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673078 2024-02-05T05:52:48.541 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673078 2024-02-05T05:52:48.541 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673078' 2024-02-05T05:52:48.541 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:48.541 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:52:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509548 2024-02-05T05:52:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509548 2024-02-05T05:52:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509548' 2024-02-05T05:52:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:52:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846595 2024-02-05T05:52:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846595 2024-02-05T05:52:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509548 3-545460846595' 2024-02-05T05:52:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:52:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182493 2024-02-05T05:52:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182493 2024-02-05T05:52:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509548 3-545460846595 4-107374182493' 2024-02-05T05:52:48.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:48.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:52:48.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846595 2024-02-05T05:52:48.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846595 2024-02-05T05:52:48.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509548 3-545460846595 4-107374182493 5-545460846595' 2024-02-05T05:52:48.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:48.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:52:48.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855439 2024-02-05T05:52:48.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855439 2024-02-05T05:52:48.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509548 3-545460846595 4-107374182493 5-545460846595 6-150323855439' 2024-02-05T05:52:48.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:49.264 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:52:49.264 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691911 2024-02-05T05:52:49.264 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691911 2024-02-05T05:52:49.264 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509548 3-545460846595 4-107374182493 5-545460846595 6-150323855439 7-171798691911' 2024-02-05T05:52:49.264 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:49.264 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:52:49.264 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528380 2024-02-05T05:52:49.264 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528380 2024-02-05T05:52:49.264 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509548 3-545460846595 4-107374182493 5-545460846595 6-150323855439 7-171798691911 8-193273528380' 2024-02-05T05:52:49.265 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:49.265 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:52:49.265 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467983 2024-02-05T05:52:49.265 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467983 2024-02-05T05:52:49.265 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509548 3-545460846595 4-107374182493 5-545460846595 6-150323855439 7-171798691911 8-193273528380 9-463856467983' 2024-02-05T05:52:49.265 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:49.265 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836598 2024-02-05T05:52:49.265 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:49.265 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:52:49.265 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836598 2024-02-05T05:52:49.265 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:49.265 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836598 2024-02-05T05:52:49.265 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836598' 2024-02-05T05:52:49.266 INFO:tasks.workunit.client.0.smithi148.stderr:271211: waiting osd.0 seq 21474836598 2024-02-05T05:52:49.266 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:52:49.266 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836599 -lt 21474836598 2024-02-05T05:52:49.266 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:49.266 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:49.266 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673078 2024-02-05T05:52:49.266 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:52:49.266 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673078 2024-02-05T05:52:49.266 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:49.266 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673078 2024-02-05T05:52:49.266 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673078' 2024-02-05T05:52:49.314 INFO:tasks.workunit.client.0.smithi148.stderr:271211: waiting tu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528379 2024-02-05T05:52:49.314 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528379 2024-02-05T05:52:49.315 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673077 2-64424509547 3-545460846594 4-107374182492 5-545460846594 6-150323855438 7-171798691910 8-193273528379' 2024-02-05T05:52:49.315 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:49.315 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:52:49.315 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467982 2024-02-05T05:52:49.315 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467982 2024-02-05T05:52:49.315 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673077 2-64424509547 3-545460846594 4-107374182492 5-545460846594 6-150323855438 7-171798691910 8-193273528379 9-463856467982' 2024-02-05T05:52:49.315 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:49.315 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836597 2024-02-05T05:52:49.315 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:49.315 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:52:49.315 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836597 2024-02-05T05:52:49.315 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:49.315 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836597 2024-02-05T05:52:49.316 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836597' 2024-02-05T05:52:49.316 INFO:tasks.workunit.client.0.smithi148.stderr:271213: waiting osd.0 seq 21474836597 2024-02-05T05:52:49.316 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:52:49.316 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836599 -lt 21474836597 2024-02-05T05:52:49.316 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:49.316 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673077 2024-02-05T05:52:49.316 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:49.316 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:52:49.316 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673077 2024-02-05T05:52:49.316 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:49.316 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673077 2024-02-05T05:52:49.316 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673077' 2024-02-05T05:52:49.316 INFO:tasks.workunit.client.0.smithi148.stderr:271213: waiting osd.1 seq 42949673077 2024-02-05T05:52:49.317 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:52:49.317 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673078 -lt 42949673077 2024-02-05T05:52:49.317 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:49.317 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509547 2024-02-05T05:52:49.317 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:49.317 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:52:50.275 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509547 2024-02-05T05:52:50.275 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:50.276 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509547 2024-02-05T05:52:50.276 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509547' 2024-02-05T05:52:50.276 INFO:tasks.workunit.client.0.smithi148.stderr:271213: waiting osd.2 seq 64424509547 2024-02-05T05:52:50.276 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:52:50.276 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509548 -lt 64424509547 2024-02-05T05:52:50.276 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:50.276 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:50.276 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-545460846594 2024-02-05T05:52:50.276 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:52:50.276 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-545460846594 2024-02-05T05:52:50.276 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:50.276 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846594 2024-02-05T05:52:50.276 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 545460846594' 2024-02-05T05:52:50.277 INFO:tasks.workunit.client.0.smithi148.stderr:271213: waiting osd.3 seq 545460846594 2024-02-05T05:52:50.277 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:52:50.277 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846595 -lt 545460846594 2024-02-05T05:52:50.277 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:50.277 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:50.277 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182492 2024-02-05T05:52:50.277 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:52:50.277 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182492 2024-02-05T05:52:50.277 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:50.277 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182492 2024-02-05T05:52:50.277 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182492' 2024-02-05T05:52:50.277 INFO:tasks.workunit.client.0.smithi148.stderr:271213: waiting osd.4 seq 107374182492 2024-02-05T05:52:50.277 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:52:50.278 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182493 -lt 107374182492 2024-02-05T05:52:50.278 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:50.278 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-545460846594 2024-02-05T05:52:50.278 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:50.278 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:52:50.278 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-545460846594 2024-02-05T05:52:50.278 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:50.278 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846594 2024-02-05T05:52:50.610 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stosd.1 seq 42949673078 2024-02-05T05:52:50.611 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:52:50.611 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673078 -lt 42949673078 2024-02-05T05:52:50.611 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:50.611 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:50.611 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509548 2024-02-05T05:52:50.611 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:52:50.611 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509548 2024-02-05T05:52:50.612 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:50.612 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509548 2024-02-05T05:52:50.612 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509548' 2024-02-05T05:52:50.612 INFO:tasks.workunit.client.0.smithi148.stderr:271211: waiting osd.2 seq 64424509548 2024-02-05T05:52:50.612 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:52:50.612 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509548 -lt 64424509548 2024-02-05T05:52:50.612 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:50.613 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-545460846595 2024-02-05T05:52:50.613 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:50.613 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:52:50.613 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-545460846595 2024-02-05T05:52:50.613 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:50.613 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846595 2024-02-05T05:52:50.613 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 545460846595' 2024-02-05T05:52:50.613 INFO:tasks.workunit.client.0.smithi148.stderr:271211: waiting osd.3 seq 545460846595 2024-02-05T05:52:50.613 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:52:50.613 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846595 -lt 545460846595 2024-02-05T05:52:50.614 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:50.614 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182493 2024-02-05T05:52:50.614 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:50.614 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:52:50.614 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182493 2024-02-05T05:52:50.614 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:50.614 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182493 2024-02-05T05:52:50.614 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182493' 2024-02-05T05:52:50.614 INFO:tasks.workunit.client.0.smithi148.stderr:271211: waiting osd.4 seq 107374182493 2024-02-05T05:52:50.614 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:52:50.614 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182493 -lt 107374182493 2024-02-05T05:52:50.614 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:51.541 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-545460846595 2024-02-05T05:52:51.541 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:51.541 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:52:51.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-545460846595 2024-02-05T05:52:51.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:51.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846595 2024-02-05T05:52:51.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 545460846595' 2024-02-05T05:52:51.542 INFO:tasks.workunit.client.0.smithi148.stderr:271211: waiting osd.5 seq 545460846595 2024-02-05T05:52:51.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:52:51.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846596 -lt 545460846595 2024-02-05T05:52:51.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:51.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855439 2024-02-05T05:52:51.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:51.543 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:52:51.544 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855439 2024-02-05T05:52:51.544 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:51.544 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855439 2024-02-05T05:52:51.544 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855439' 2024-02-05T05:52:51.544 INFO:tasks.workunit.client.0.smithi148.stderr:271211: waiting osd.6 seq 150323855439 2024-02-05T05:52:51.545 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:52:51.545 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855439 -lt 150323855439 2024-02-05T05:52:51.545 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:51.545 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691911 2024-02-05T05:52:51.545 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:51.545 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:52:51.546 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691911 2024-02-05T05:52:51.546 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:51.546 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691911 2024-02-05T05:52:51.546 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691911' 2024-02-05T05:52:51.546 INFO:tasks.workunit.client.0.smithi148.stderr:271211: waiting osd.7 seq 171798691911 2024-02-05T05:52:51.546 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:52:51.547 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691912 -lt 171798691911 2024-02-05T05:52:51.547 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:51.547 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528380 2024-02-05T05:52:51.547 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:51.547 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:52:51.547 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8ats: echo 'waiting osd.5 seq 545460846594' 2024-02-05T05:52:51.548 INFO:tasks.workunit.client.0.smithi148.stderr:271213: waiting osd.5 seq 545460846594 2024-02-05T05:52:51.548 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:52:51.548 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846596 -lt 545460846594 2024-02-05T05:52:51.548 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:51.548 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855438 2024-02-05T05:52:51.549 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:51.549 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:52:51.549 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855438 2024-02-05T05:52:51.549 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:51.549 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855438 2024-02-05T05:52:51.549 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855438' 2024-02-05T05:52:51.550 INFO:tasks.workunit.client.0.smithi148.stderr:271213: waiting osd.6 seq 150323855438 2024-02-05T05:52:51.550 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:52:51.550 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855439 -lt 150323855438 2024-02-05T05:52:51.550 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:51.550 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691910 2024-02-05T05:52:51.550 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:51.551 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:52:51.551 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691910 2024-02-05T05:52:51.551 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:51.551 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691910 2024-02-05T05:52:51.551 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691910' 2024-02-05T05:52:51.551 INFO:tasks.workunit.client.0.smithi148.stderr:271213: waiting osd.7 seq 171798691910 2024-02-05T05:52:51.552 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:52:51.552 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691912 -lt 171798691910 2024-02-05T05:52:51.552 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:51.552 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528379 2024-02-05T05:52:51.552 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:51.553 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:52:51.553 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528379 2024-02-05T05:52:51.553 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:51.553 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528379 2024-02-05T05:52:51.553 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528379' 2024-02-05T05:52:51.553 INFO:tasks.workunit.client.0.smithi148.stderr:271213: waiting osd.8 seq 193273528379 2024-02-05T05:52:51.554 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:52:51.554 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528380 -lt 193273528379 2024-02-05T05:52:52.933 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:52.934 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-463856467982 2024-02-05T05:52:52.934 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:52.934 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:52:52.934 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-463856467982 2024-02-05T05:52:52.934 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:52.934 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467982 2024-02-05T05:52:52.934 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 463856467982' 2024-02-05T05:52:52.934 INFO:tasks.workunit.client.0.smithi148.stderr:271213: waiting osd.9 seq 463856467982 2024-02-05T05:52:52.934 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:52:52.934 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467983 -lt 463856467982 2024-02-05T05:52:52.934 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:52:52.934 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:52:52.934 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:52:52.935 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:52:52.935 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:52:52.935 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:52:52.935 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:52:52.935 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:52:52.935 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:52:52.935 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:52:52.935 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:52:52.935 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:52:52.935 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:52:52.935 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:52:52.935 INFO:tasks.workunit.client.0.smithi148.stderr:271213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:52:52.935 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:52:52.936 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:52:52.936 INFO:tasks.workunit.client.0.smithi148.stderr:271213: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:52:52.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:52:53.305 INFO:tasks.workunit.client.0.smithi148.stderr:-193273528380 2024-02-05T05:52:53.305 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:53.305 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528380 2024-02-05T05:52:53.305 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528380' 2024-02-05T05:52:53.305 INFO:tasks.workunit.client.0.smithi148.stderr:271211: waiting osd.8 seq 193273528380 2024-02-05T05:52:53.305 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:52:53.305 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528380 -lt 193273528380 2024-02-05T05:52:53.305 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-463856467983 2024-02-05T05:52:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:52:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-463856467983 2024-02-05T05:52:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467983 2024-02-05T05:52:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 463856467983' 2024-02-05T05:52:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:271211: waiting osd.9 seq 463856467983 2024-02-05T05:52:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:52:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467983 -lt 463856467983 2024-02-05T05:52:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:52:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:52:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:52:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:271211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:271211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:52:53.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:52:53.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:52:53.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 271209 2024-02-05T05:52:53.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:52:53.308 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:52:53.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:52:53.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:52:53.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:52:53.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:52:53.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:52:53.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:52:53.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:52:53.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:52:53.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:52:53.388 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:52:53.388 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:52:53.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:52:53.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:52:53.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:52:53.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:52:53.704 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:52:53.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:52:53.706 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:52:53.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:52:53.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:52:53.708 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:52:53.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:52:53.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:52:53.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:52:53.711 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:52:53.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:52:53.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:52:53.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:52:53.785 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:52:53.785 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:53.786 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:52:53.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:52:53.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:52:53.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:52:53.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:52:53.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:52:53.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:52:53.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:52:53.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:52:53.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:52:53.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:52:53.788 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:52:53.788 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:52:53.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:52:53.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:52:53.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:52:53.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:52:53.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:52:53.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:52:53.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:52:53.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:52:53.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:52:53.793 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:52:53.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:52:53.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:52:53.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:52:53.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:52:53.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:52:53.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:52:53.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:52:53.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:52:53.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:53.801 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:52:53.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:52:53.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:52:53.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:52:53.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:52:53.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:52:53.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:53.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:52:53.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T05:52:53.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:52:53.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:52:53.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:52:53.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:52:53.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-02-05T05:52:53.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-02-05T05:52:53.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-02-05T05:52:53.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-02-05T05:52:53.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:52:53.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:52:53.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:52:53.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:52:53.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:52:53.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-02-05T05:52:54.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:52:54.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:52:54.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:52:54.181 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:52:54.181 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:54.181 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:52:54.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:54.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T05:52:54.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:52:54.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:52:54.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:52:54.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:52:54.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:52:54.428 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:52:54.428 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:52:54.428 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:52:54.428 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:52:54.428 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:52:54.428 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:54.429 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:52:54.429 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:52:54.429 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:52:54.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T05:52:54.542 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:52:54.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:52:54.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:52:54.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:52:54.544 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:52:54.544 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:52:54.544 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:52:54.544 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:52:54.544 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:52:54.544 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:54.544 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:52:54.544 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:52:54.545 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:52:54.545 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T05:52:54.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-02-05T05:52:54.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:52:54.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:52:54.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:52:54.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:52:54.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:52:54.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:52:55.516 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:52:55.516 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:52:55.516 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:52:55.517 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:52:55.517 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:55.517 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:52:55.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:55.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:52:55.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:52:55.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-02-05T05:52:55.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:52:55.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:52:55.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:52:55.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:52:55.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:52:55.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:52:55.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:52:55.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:52:55.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:52:55.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:52:55.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:52:55.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:52:55.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:52:55.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:52:55.556 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:52:55.557 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:52:55.557 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:52:55.557 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:52:55.557 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:55.557 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:52:55.557 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:55.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:52:55.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:52:55.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:52:55.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:52:55.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:52:55.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:52:55.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:52:55.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:52:55.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:52:55.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:52:55.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:52:55.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:52:55.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:52:55.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:52:55.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9f6922b6-3465-4004-a8b6-c4ff14b4e2de 2024-02-05T05:52:55.559 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 9f6922b6-3465-4004-a8b6-c4ff14b4e2de 2024-02-05T05:52:55.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9f6922b6-3465-4004-a8b6-c4ff14b4e2de' 2024-02-05T05:52:55.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:52:55.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3d8BlC1dhIxAA6YrG18RkdEmzhhgwZOIF3A== 2024-02-05T05:52:55.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3d8BlC1dhIxAA6YrG18RkdEmzhhgwZOIF3A=="}' 2024-02-05T05:52:55.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9f6922b6-3465-4004-a8b6-c4ff14b4e2de -i td/osd-scrub-repair/0/new.json 2024-02-05T05:52:55.847 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:52:55.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:52:55.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3d8BlC1dhIxAA6YrG18RkdEmzhhgwZOIF3A== --osd-uuid 9f6922b6-3465-4004-a8b6-c4ff14b4e2de 2024-02-05T05:52:55.897 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:52:55.912+0000 7f0b9423f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:55.912 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:52:55.928+0000 7f0b9423f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:55.913 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:52:55.928+0000 7f0b9423f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:55.914 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:52:55.928+0000 7f0b9423f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:52:55.914 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:52:55.928+0000 7f0b9423f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:52:55.914 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:52:55.928+0000 7f0b9423f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:52:56.013 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:52:56.028+0000 7f0b9423f640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:53:02.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:53:02.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:53:02.293 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:53:02.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:53:02.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:53:02.680 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:53:02.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:53:02.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:53:02.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:53:02.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:53:02.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:53:02.713 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:02.728+0000 7fcfbdbc1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:02.736 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:02.752+0000 7fcfbdbc1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:02.738 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:02.752+0000 7fcfbdbc1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:02.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:53:02.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:53:02.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:53:02.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:53:02.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:53:02.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:02.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:53:02.990 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:53:02.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:02.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:03.286 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:03.300+0000 7fcfbdbc1640 -1 Falling back to public interface 2024-02-05T05:53:03.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:04.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:04.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:04.319 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:53:04.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:53:04.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:04.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:04.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:05.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:05.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:05.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:53:05.633 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:53:05.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:05.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:05.930 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:05.944+0000 7fcfbdbc1640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:53:05.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:06.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:06.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:06.935 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:53:06.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:53:06.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:06.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:07.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:08.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:08.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:08.279 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:53:08.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:53:08.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:08.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:08.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:09.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:09.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:09.581 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:53:09.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:53:09.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:09.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:09.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:10.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:10.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:10.920 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:53:10.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:53:10.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:10.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:11.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:12.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:12.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:12.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:53:12.224 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:53:12.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:12.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:12.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:13.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:13.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:13.529 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:53:13.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:53:13.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:13.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:14.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:15.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:15.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:15.063 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:53:15.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:53:15.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:15.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:15.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:16.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:16.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:16.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:53:16.386 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:53:16.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:16.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:16.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:17.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:17.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:17.706 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:53:17.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:53:17.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:17.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:18.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:19.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:19.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:19.097 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:53:19.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:53:19.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:19.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:19.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:20.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:20.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:20.405 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:53:20.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:53:20.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:20.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:20.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:21.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:21.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:21.719 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:53:21.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:53:21.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:21.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:22.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:23.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:23.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:23.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:53:23.485 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:53:23.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:23.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:24.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:25.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:25.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:25.132 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:53:25.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:53:25.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:25.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:25.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:26.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:26.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:26.500 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:53:26.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:53:26.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:26.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:27.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:28.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:28.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:28.323 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:53:28.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:53:28.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:28.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:28.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:29.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:29.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:29.841 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:53:29.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:53:29.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:29.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:30.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:31.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:31.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:31.821 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:53:31.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:53:31.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:31.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:32.141 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/169337669,v1:127.0.0.1:6803/169337669] [v2:127.0.0.1:6804/169337669,v1:127.0.0.1:6805/169337669] exists,up 9f6922b6-3465-4004-a8b6-c4ff14b4e2de 2024-02-05T05:53:32.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:53:32.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:53:32.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:53:32.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-02-05T05:53:32.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:53:32.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:53:32.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:53:32.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:53:32.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:53:32.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:53:32.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:53:32.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:53:32.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:53:32.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:53:32.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:53:32.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:53:32.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:53:32.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:53:32.145 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:53:32.145 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:53:32.145 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:53:32.145 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:53:32.145 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:53:32.145 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:53:32.146 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:53:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:53:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:53:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:53:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:53:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:53:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:53:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:53:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:53:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:53:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:53:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:53:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:53:32.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:53:32.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:53:32.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cdffc5a6-b213-4085-a385-d511a1938e43 2024-02-05T05:53:32.148 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 cdffc5a6-b213-4085-a385-d511a1938e43 2024-02-05T05:53:32.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cdffc5a6-b213-4085-a385-d511a1938e43' 2024-02-05T05:53:32.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:53:32.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDcd8Blj4Q0CxAAvMOKWfpCvWtjmuXmTPYpTQ== 2024-02-05T05:53:32.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDcd8Blj4Q0CxAAvMOKWfpCvWtjmuXmTPYpTQ=="}' 2024-02-05T05:53:32.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cdffc5a6-b213-4085-a385-d511a1938e43 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:53:32.552 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:53:32.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:53:32.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDcd8Blj4Q0CxAAvMOKWfpCvWtjmuXmTPYpTQ== --osd-uuid cdffc5a6-b213-4085-a385-d511a1938e43 2024-02-05T05:53:32.609 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:32.623+0000 7f4413220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:32.654 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:32.671+0000 7f4413220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:32.656 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:32.671+0000 7f4413220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:32.656 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:32.671+0000 7f4413220640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:53:32.656 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:32.671+0000 7f4413220640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:53:32.656 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:32.671+0000 7f4413220640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:53:32.748 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:32.763+0000 7f4413220640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:53:39.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:53:39.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:53:39.550 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:53:39.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:53:39.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:53:39.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:53:39.922 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:53:39.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:53:39.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:53:39.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:53:39.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:53:39.949 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:39.963+0000 7fc0f4954640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:39.951 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:39.967+0000 7fc0f4954640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:39.952 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:39.967+0000 7fc0f4954640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:40.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:53:40.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:53:40.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:53:40.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:53:40.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:53:40.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:40.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:53:40.299 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:53:40.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:40.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:53:40.489 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:40.503+0000 7fc0f4954640 -1 Falling back to public interface 2024-02-05T05:53:40.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:41.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:41.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:41.611 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:53:41.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:53:41.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:41.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:53:41.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:42.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:42.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:42.918 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:53:42.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:53:42.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:42.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:53:43.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:43.244 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:53:43.259+0000 7fc0f4954640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:53:44.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:44.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:44.236 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:53:44.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:53:44.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:44.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:53:44.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:45.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:45.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:45.710 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:53:45.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:53:45.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:45.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:53:46.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:47.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:47.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:47.021 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:53:47.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:53:47.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:47.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:53:47.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:48.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:48.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:48.348 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:53:48.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:53:48.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:48.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:53:48.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:49.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:49.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:49.728 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:53:49.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:53:49.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:49.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:53:50.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:51.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:51.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:51.267 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:53:51.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:53:51.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:51.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:53:51.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:52.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:52.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:52.690 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:53:52.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:53:52.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:52.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:53:53.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:54.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:54.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:54.098 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:53:54.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:53:54.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:54.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:53:54.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:55.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:55.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:55.452 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:53:55.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:53:55.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:55.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:53:55.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:56.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:56.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:56.768 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:53:56.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:53:56.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:56.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:53:57.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:58.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:58.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:58.140 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:53:58.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:53:58.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:58.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:53:58.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:59.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:59.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:59.995 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:53:59.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:53:59.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:59.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:00.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:01.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:01.503 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:54:01.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:01.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:54:01.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:02.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:03.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:03.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:03.048 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:54:03.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:54:03.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:03.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:03.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:04.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:04.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:04.774 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:54:04.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:54:04.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:04.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:05.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:06.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:06.150 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:54:06.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:06.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:54:06.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:06.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:06.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:07.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:07.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:07.583 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:54:07.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:54:07.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:07.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:08.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:09.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:09.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:09.456 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:54:09.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:54:09.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:09.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:09.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:10.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:10.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:10.834 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T05:54:10.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T05:54:10.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:10.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:11.146 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3565299296,v1:127.0.0.1:6811/3565299296] [v2:127.0.0.1:6812/3565299296,v1:127.0.0.1:6813/3565299296] exists,up cdffc5a6-b213-4085-a385-d511a1938e43 2024-02-05T05:54:11.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:54:11.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:54:11.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:54:11.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-02-05T05:54:11.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:54:11.487 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T05:54:11.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:54:11.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:54:11.946 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T05:54:11.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:54:12.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:54:14.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-02-05T05:54:14.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:54:14.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:54:14.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:54:14.875 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:54:14.876 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:54:14.877 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:54:14.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:54:14.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:54:14.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:54:15.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:54:15.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:54:15.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:54:15.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:54:15.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:54:15.007 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:54:15.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:54:15.329 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T05:54:15.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:54:15.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:54:15.329 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:54:15.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-02-05T05:54:15.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-02-05T05:54:15.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-02-05T05:54:15.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:54:15.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:54:15.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672962 2024-02-05T05:54:15.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672962 2024-02-05T05:54:15.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-42949672962' 2024-02-05T05:54:15.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:54:15.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-02-05T05:54:15.750 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:54:15.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:54:15.751 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-02-05T05:54:15.751 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:54:15.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-02-05T05:54:15.753 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836491 2024-02-05T05:54:15.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-02-05T05:54:15.753 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:54:16.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836491 2024-02-05T05:54:16.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:54:17.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:54:17.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:54:17.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-02-05T05:54:17.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:54:17.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672962 2024-02-05T05:54:17.382 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:54:17.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:54:17.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672962 2024-02-05T05:54:17.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:54:17.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672962 2024-02-05T05:54:17.386 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672962 2024-02-05T05:54:17.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2024-02-05T05:54:17.386 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:54:17.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672963 -lt 42949672962 2024-02-05T05:54:17.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:54:17.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:54:17.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:54:18.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:54:18.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:54:18.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:54:18.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:54:18.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:54:18.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:54:18.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:54:18.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:54:18.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:54:18.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:54:18.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:54:18.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:54:18.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:54:18.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:54:18.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:54:18.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-02-05T05:54:18.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:54:18.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-02-05T05:54:18.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T05:54:18.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:54:18.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:54:18.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:54:19.208 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T05:54:19.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:54:19.777 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T05:54:19.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:54:19.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:54:19.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:20.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-02-05T05:54:20.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-02-05T05:54:20.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:54:20.143 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-02-05T05:54:20.143 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-02-05T05:54:20.143 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:54:20.143 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-02-05T05:54:20.143 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:54:20.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:54:20.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-02-05T05:54:20.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:54:20.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-02-05T05:54:20.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:54:20.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-02-05T05:54:20.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-02-05T05:54:20.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:54:20.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:54:20.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:54:20.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:54:20.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:54:20.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:54:20.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:54:20.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:54:20.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:54:20.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:54:20.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:54:20.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:54:20.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:54:20.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:54:20.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:54:20.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:54:21.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:54:21.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:54:21.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:54:21.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:54:21.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:54:21.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:54:21.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:54:21.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-02-05T05:54:22.875 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-02-05T05:54:23.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:54:23.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:54:23.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:54:23.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:54:23.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:54:23.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:54:23.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:54:23.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:54:23.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:54:23.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:54:23.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:54:23.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:54:23.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:54:23.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:54:23.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:54:23.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:54:23.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:54:23.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:54:23.411 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:54:23.411 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:54:23.411 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:54:23.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:54:23.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:54:23.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:54:23.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:54:23.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:54:23.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:54:23.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:54:23.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:54:23.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:54:23.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:54:23.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:54:23.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:54:23.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:54:23.413 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:54:23.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:54:23.413 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:54:23.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:54:23.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:54:23.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:54:23.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:54:23.446 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:23.462+0000 7f24582c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:23.448 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:23.466+0000 7f24582c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:23.449 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:23.466+0000 7f24582c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:23.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:54:23.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:54:23.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:54:23.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:54:23.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:54:23.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:23.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:54:23.737 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:54:23.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:23.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:23.984 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:24.002+0000 7f24582c4640 -1 Falling back to public interface 2024-02-05T05:54:24.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:25.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:25.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:25.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:54:25.072 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:54:25.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:25.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:25.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:26.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:26.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:26.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:54:26.372 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:54:26.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:26.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:26.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:26.984 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:27.002+0000 7f24582c4640 -1 osd.0 20 log_to_monitors true 2024-02-05T05:54:27.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:27.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:27.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:54:27.700 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:54:27.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:27.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:28.095 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/1814599943,v1:127.0.0.1:6803/1814599943] [v2:127.0.0.1:6804/1814599943,v1:127.0.0.1:6805/1814599943] exists,up 9f6922b6-3465-4004-a8b6-c4ff14b4e2de 2024-02-05T05:54:28.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:54:28.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:54:28.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:54:28.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:54:28.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:54:28.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:54:28.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:54:28.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:54:28.097 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:54:28.097 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:54:28.097 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:54:28.097 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:54:28.097 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:54:28.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:54:28.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:54:28.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:54:28.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:54:28.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:54:28.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:54:28.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:54:28.553 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T05:54:28.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:54:28.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:54:28.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:54:28.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-02-05T05:54:28.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-02-05T05:54:28.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-02-05T05:54:28.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:54:28.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:54:28.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-02-05T05:54:28.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-02-05T05:54:28.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672966' 2024-02-05T05:54:28.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:54:28.883 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-02-05T05:54:28.883 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:54:28.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:54:28.884 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-02-05T05:54:28.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:54:28.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-02-05T05:54:28.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-02-05T05:54:28.887 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 107374182402 2024-02-05T05:54:28.887 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:54:29.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-02-05T05:54:29.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:54:30.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:54:30.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:54:30.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-02-05T05:54:30.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:54:30.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-02-05T05:54:30.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:54:30.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:54:30.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-02-05T05:54:30.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:54:30.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-02-05T05:54:30.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-02-05T05:54:30.517 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672966 2024-02-05T05:54:30.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:54:30.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672966 2024-02-05T05:54:30.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:54:30.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:54:30.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:54:31.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:54:31.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:54:31.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:54:31.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:54:31.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:54:31.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:54:31.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:54:31.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:54:31.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:54:31.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:54:31.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:54:31.519 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:54:31.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:54:31.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:54:31.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:54:31.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-02-05T05:54:31.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-02-05T05:54:31.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:54:31.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:54:31.901 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-02-05T05:54:32.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-02-05T05:54:32.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-02-05T05:54:32.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-02-05T05:54:32.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-02-05T05:54:32.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:54:32.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:54:32.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:54:32.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:54:32.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:54:11.889941+0000 2024-02-05T05:54:32.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-02-05T05:54:32.916 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 1.3 on osd.1 to repair 2024-02-05T05:54:32.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-02-05T05:54:11.889941+0000 2024-02-05T05:54:32.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-02-05T05:54:32.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:54:11.889941+0000 2024-02-05T05:54:32.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:54:32.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:54:32.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:54:32.934 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:54:32.934 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:54:32.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:54:32.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:54:32.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:54:33.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:54:11.889941+0000 '>' 2024-02-05T05:54:11.889941+0000 2024-02-05T05:54:33.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:54:34.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:54:34.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:54:34.216 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:54:34.216 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:54:34.216 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:54:34.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:54:34.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:54:34.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:54:11.889941+0000 '>' 2024-02-05T05:54:11.889941+0000 2024-02-05T05:54:34.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:54:35.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:54:35.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:54:35.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:54:35.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:54:35.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:54:35.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:54:35.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:54:35.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:54:11.889941+0000 '>' 2024-02-05T05:54:11.889941+0000 2024-02-05T05:54:35.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:54:36.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:54:36.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:54:36.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:54:36.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:54:36.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:54:36.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:54:36.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:54:37.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:54:33.151022+0000 '>' 2024-02-05T05:54:11.889941+0000 2024-02-05T05:54:37.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:54:37.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:54:37.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:54:37.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:54:37.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:54:37.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:54:37.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:54:37.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:54:37.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:54:37.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:54:37.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:54:37.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:54:37.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:54:37.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:54:37.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:54:37.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:54:37.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:54:37.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:54:37.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:54:37.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:54:37.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:54:37.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:54:37.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:54:37.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:54:37.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:54:39.050 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-02-05T05:54:39.050 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:54:39.050 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:54:39.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:54:39.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:54:39.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:54:39.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:54:39.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:54:39.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:54:39.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:54:39.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:54:39.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:54:39.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:54:39.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:54:39.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:54:39.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:54:39.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:54:39.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:54:39.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:54:39.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:54:39.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:54:39.352 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:54:39.352 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:54:39.352 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:54:39.352 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:54:39.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:54:39.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:54:39.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:54:39.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:54:39.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:54:39.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:54:39.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:54:39.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:54:39.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:54:39.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:54:39.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:54:39.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:54:39.354 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:54:39.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:54:39.354 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:54:39.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:54:39.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:54:39.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:54:39.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:54:39.393 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:39.409+0000 7f7e79dc6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:39.396 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:39.413+0000 7f7e79dc6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:39.397 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:39.413+0000 7f7e79dc6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:39.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:54:39.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:54:39.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:54:39.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:54:39.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:54:39.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:39.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:54:39.666 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:54:39.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:39.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:39.933 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:39.949+0000 7f7e79dc6640 -1 Falling back to public interface 2024-02-05T05:54:39.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:40.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:40.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:40.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:54:40.980 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:54:40.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:40.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:41.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:42.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:42.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:42.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:54:42.312 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:54:42.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:42.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:42.354 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:42.369+0000 7f7e79dc6640 -1 osd.1 26 log_to_monitors true 2024-02-05T05:54:42.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:43.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:43.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:43.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:54:43.678 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:54:43.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:43.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:43.993 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/1300952359,v1:127.0.0.1:6811/1300952359] [v2:127.0.0.1:6812/1300952359,v1:127.0.0.1:6813/1300952359] exists,up cdffc5a6-b213-4085-a385-d511a1938e43 2024-02-05T05:54:43.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:54:43.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:54:43.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:54:43.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:54:43.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:54:43.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:54:43.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:54:43.994 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:54:43.994 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:54:43.994 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:54:43.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:54:43.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:54:43.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:54:44.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:54:44.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:54:44.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:54:44.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:54:44.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:54:44.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:54:44.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:54:44.381 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T05:54:44.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:54:44.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:54:44.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:54:44.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182406 2024-02-05T05:54:44.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182406 2024-02-05T05:54:44.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406' 2024-02-05T05:54:44.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:54:44.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:54:44.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-02-05T05:54:44.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-02-05T05:54:44.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-128849018882' 2024-02-05T05:54:44.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:54:44.750 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182406 2024-02-05T05:54:44.750 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:54:44.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:54:44.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182406 2024-02-05T05:54:44.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:54:44.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182406 2024-02-05T05:54:44.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2024-02-05T05:54:44.756 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 107374182406 2024-02-05T05:54:44.756 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:54:45.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182406 2024-02-05T05:54:45.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:54:46.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:54:46.063 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:54:46.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182406 2024-02-05T05:54:46.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:54:46.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018882 2024-02-05T05:54:46.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:54:46.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:54:46.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018882 2024-02-05T05:54:46.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:54:46.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-02-05T05:54:46.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-02-05T05:54:46.454 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 128849018882 2024-02-05T05:54:46.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:54:46.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-02-05T05:54:46.770 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:54:46.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:54:46.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:54:47.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:54:47.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:54:47.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:54:47.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:54:47.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:54:47.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:54:47.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:54:47.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:54:47.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:54:47.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:54:47.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:54:47.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:54:47.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:54:47.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:54:47.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:54:47.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:54:47.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:54:47.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-02-05T05:54:47.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-02-05T05:54:47.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:54:47.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-02-05T05:54:47.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:54:48.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-02-05T05:54:48.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:54:48.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-02-05T05:54:48.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-02-05T05:54:48.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:54:48.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:54:48.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:54:48.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:54:48.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:54:48.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:54:48.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:54:48.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:54:48.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:54:48.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:54:48.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:54:48.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:54:48.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:54:48.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:54:48.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:54:48.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:54:49.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:54:49.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:54:49.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:54:49.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:54:49.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:54:49.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:54:49.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:54:49.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:54:50.945 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-02-05T05:54:51.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:54:51.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:54:51.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:54:51.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:54:51.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:54:51.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:54:51.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:54:51.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:54:51.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:54:51.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:54:51.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:54:51.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:54:51.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:54:51.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:54:51.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:54:51.490 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:54:51.490 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:54:51.490 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:54:51.492 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:54:51.492 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:54:51.492 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:54:51.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:54:51.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:54:51.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:54:51.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:54:51.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:54:51.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:54:51.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:54:51.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:54:51.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:54:51.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:54:51.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:54:51.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:54:51.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:54:51.493 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:54:51.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:54:51.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:54:51.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:54:51.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:54:51.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:54:51.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:54:51.527 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:51.545+0000 7fd7064d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:51.530 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:51.549+0000 7fd7064d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:51.531 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:51.549+0000 7fd7064d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:51.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:54:51.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:54:51.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:54:51.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:54:51.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:54:51.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:51.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:54:51.812 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:54:51.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:51.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:52.065 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:52.081+0000 7fd7064d4640 -1 Falling back to public interface 2024-02-05T05:54:52.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:53.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:53.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:53.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:54:53.151 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:54:53.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:53.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:53.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:54.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:54.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:54.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:54:54.467 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:54:54.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:54.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:54.834 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:54:54.853+0000 7fd7064d4640 -1 osd.1 31 log_to_monitors true 2024-02-05T05:54:54.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:55.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:55.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:55.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:54:55.864 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:54:55.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:55.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:56.370 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6810/2274770638,v1:127.0.0.1:6811/2274770638] [v2:127.0.0.1:6812/2274770638,v1:127.0.0.1:6813/2274770638] exists,up cdffc5a6-b213-4085-a385-d511a1938e43 2024-02-05T05:54:56.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:54:56.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:54:56.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:54:56.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:54:56.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:54:56.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:54:56.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:54:56.372 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:54:56.372 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:54:56.372 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:54:56.372 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:54:56.372 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:54:56.372 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:54:56.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:54:56.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:54:56.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:54:56.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:54:56.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:54:56.490 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:54:56.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:54:56.870 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T05:54:56.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:54:56.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:54:56.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:54:57.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182409 2024-02-05T05:54:57.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182409 2024-02-05T05:54:57.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182409' 2024-02-05T05:54:57.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:54:57.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:54:57.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-02-05T05:54:57.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-02-05T05:54:57.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182409 1-150323855362' 2024-02-05T05:54:57.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:54:57.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182409 2024-02-05T05:54:57.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:54:57.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:54:57.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182409 2024-02-05T05:54:57.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:54:57.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182409 2024-02-05T05:54:57.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182409' 2024-02-05T05:54:57.208 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 107374182409 2024-02-05T05:54:57.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:54:57.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182408 -lt 107374182409 2024-02-05T05:54:57.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:54:58.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:54:58.516 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:54:58.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182410 -lt 107374182409 2024-02-05T05:54:58.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:54:58.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-02-05T05:54:58.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:54:58.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:54:58.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-02-05T05:54:58.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:54:58.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-02-05T05:54:58.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-02-05T05:54:58.856 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 150323855362 2024-02-05T05:54:58.856 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:54:59.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-02-05T05:54:59.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:54:59.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:54:59.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:54:59.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:54:59.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:54:59.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:54:59.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:54:59.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:54:59.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:54:59.568 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:54:59.568 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:54:59.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:54:59.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:54:59.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:54:59.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:55:00.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:55:00.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:55:00.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:55:00.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-02-05T05:55:00.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-02-05T05:55:00.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:55:00.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-02-05T05:55:00.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:55:00.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-02-05T05:55:00.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-02-05T05:55:00.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-02-05T05:55:00.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-02-05T05:55:00.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:55:00.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:55:00.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:55:00.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:55:00.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:54:33.151022+0000 2024-02-05T05:55:00.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-02-05T05:55:01.171 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 1.3 on osd.1 to repair 2024-02-05T05:55:01.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-02-05T05:54:33.151022+0000 2024-02-05T05:55:01.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-02-05T05:55:01.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:54:33.151022+0000 2024-02-05T05:55:01.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:55:01.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:55:01.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:55:01.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:55:01.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:55:01.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:55:01.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:55:01.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:55:01.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:54:33.151022+0000 '>' 2024-02-05T05:54:33.151022+0000 2024-02-05T05:55:01.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:55:02.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:55:02.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:55:02.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:55:02.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:55:02.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:55:02.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:55:02.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:55:02.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:54:33.151022+0000 '>' 2024-02-05T05:54:33.151022+0000 2024-02-05T05:55:02.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:55:03.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:55:03.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:55:03.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:55:03.801 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:55:03.801 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:55:03.801 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:55:03.801 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:55:04.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:54:33.151022+0000 '>' 2024-02-05T05:54:33.151022+0000 2024-02-05T05:55:04.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:55:05.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:55:05.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:55:05.116 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:55:05.117 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:55:05.117 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:55:05.117 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:55:05.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:55:05.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:54:33.151022+0000 '>' 2024-02-05T05:54:33.151022+0000 2024-02-05T05:55:05.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:55:06.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:55:06.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:55:06.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:55:06.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:55:06.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:55:06.435 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:55:06.435 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:55:06.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:55:01.816408+0000 '>' 2024-02-05T05:54:33.151022+0000 2024-02-05T05:55:06.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:55:06.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:55:06.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:55:06.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:55:06.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:55:06.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:55:06.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:55:06.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:55:06.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:55:06.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:55:06.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:55:06.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:55:06.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:55:06.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:55:06.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:55:06.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:55:06.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:55:08.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:55:08.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:55:08.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:08.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:08.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:55:08.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:08.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:55:08.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:55:09.303 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-02-05T05:55:09.304 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T05:55:09.304 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T05:55:09.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:55:09.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:55:09.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:55:09.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:55:09.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:55:09.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:55:09.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:55:09.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:55:09.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:55:09.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:55:09.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:55:09.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:55:09.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:55:09.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:55:09.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:55:09.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:55:09.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:55:09.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:55:09.842 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:55:09.842 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:09.842 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:55:09.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:55:09.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:55:09.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:55:09.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:55:09.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:55:09.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:55:09.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:55:09.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:55:09.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:55:09.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:55:09.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:55:09.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:55:09.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:55:09.844 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:55:09.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:55:09.845 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:55:09.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:55:09.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:55:09.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:55:09.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:55:09.878 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:09.896+0000 7f6a2d2c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:09.881 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:09.896+0000 7f6a2d2c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:09.884 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:09.900+0000 7f6a2d2c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:10.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:55:10.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:55:10.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:55:10.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:55:10.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:55:10.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:10.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:55:10.162 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:55:10.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:10.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:55:10.435 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:10.452+0000 7f6a2d2c0640 -1 Falling back to public interface 2024-02-05T05:55:10.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:11.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:11.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:11.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:55:11.483 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:55:11.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:11.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:55:11.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:12.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:12.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:12.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:55:12.787 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:55:12.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:12.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:55:13.102 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:13.120+0000 7f6a2d2c0640 -1 osd.1 37 log_to_monitors true 2024-02-05T05:55:13.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:14.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:14.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:14.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:55:14.107 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:55:14.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:14.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:55:14.434 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [35,37) [v2:127.0.0.1:6810/273390080,v1:127.0.0.1:6811/273390080] [v2:127.0.0.1:6812/273390080,v1:127.0.0.1:6813/273390080] exists,up cdffc5a6-b213-4085-a385-d511a1938e43 2024-02-05T05:55:14.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:55:14.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:55:14.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:55:14.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:55:14.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:55:14.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:55:14.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:55:14.435 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:55:14.436 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:55:14.437 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:55:14.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:55:14.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:55:14.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:55:14.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:55:14.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:55:14.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:55:14.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:55:14.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:55:14.545 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:55:14.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:55:14.871 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T05:55:14.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:55:14.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:55:14.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:55:15.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182414 2024-02-05T05:55:15.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182414 2024-02-05T05:55:15.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182414' 2024-02-05T05:55:15.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:55:15.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:55:15.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-02-05T05:55:15.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-02-05T05:55:15.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182414 1-176093659138' 2024-02-05T05:55:15.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:55:15.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182414 2024-02-05T05:55:15.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:55:15.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:55:15.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182414 2024-02-05T05:55:15.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:55:15.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182414 2024-02-05T05:55:15.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182414' 2024-02-05T05:55:15.156 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 107374182414 2024-02-05T05:55:15.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:55:15.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182413 -lt 107374182414 2024-02-05T05:55:15.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:55:16.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:55:16.461 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:55:16.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182414 -lt 107374182414 2024-02-05T05:55:16.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:55:16.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-176093659138 2024-02-05T05:55:16.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:55:16.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:55:16.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-176093659138 2024-02-05T05:55:16.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:55:16.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-02-05T05:55:16.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2024-02-05T05:55:16.779 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 176093659138 2024-02-05T05:55:16.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:55:17.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-02-05T05:55:17.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:55:17.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:55:17.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:55:17.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:55:17.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:55:17.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:55:17.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:55:17.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:55:17.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:55:17.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:55:17.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:55:17.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:55:17.876 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:55:17.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:55:17.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:55:18.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:55:18.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:55:18.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:55:18.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:55:18.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:55:18.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:55:18.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:55:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:55:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:55:18.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:55:18.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:55:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:55:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:55:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:55:18.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:55:18.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:55:18.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:55:18.591 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:55:18.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:55:18.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:55:18.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:55:18.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:55:18.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:55:18.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:55:18.596 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:55:18.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:55:18.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:55:18.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:55:18.616 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:55:18.616 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:18.616 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:55:18.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:55:18.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:55:18.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:55:18.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:55:18.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:55:18.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:55:18.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:55:18.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:55:18.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:55:18.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:55:18.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:55:18.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:55:18.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:55:18.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:55:18.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:55:18.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:55:18.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:55:18.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:55:18.623 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:55:18.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:55:18.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:55:18.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:55:18.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:55:18.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:55:18.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:55:18.627 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:55:18.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:55:18.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:55:18.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:55:18.631 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:55:18.631 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:18.631 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:55:18.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T05:55:18.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:55:18.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:55:18.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:55:18.634 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:55:18.634 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:18.634 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:55:18.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T05:55:18.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:55:18.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:55:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:55:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:55:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-02-05T05:55:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-02-05T05:55:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-02-05T05:55:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-02-05T05:55:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-02-05T05:55:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-02-05T05:55:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-02-05T05:55:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:55:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:55:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:55:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:55:18.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:55:18.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:55:19.051 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:55:19.051 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:55:19.051 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:55:19.052 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:55:19.052 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:19.052 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:55:19.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:55:19.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:55:19.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:55:19.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:55:19.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:55:19.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:55:19.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:55:19.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:55:19.273 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:55:19.273 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:55:19.273 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:55:19.274 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:55:19.274 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:19.274 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:55:19.274 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:55:19.275 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:55:19.275 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T05:55:19.389 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:55:19.389 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:55:19.389 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:55:19.390 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:55:19.390 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:55:19.390 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:55:19.390 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:55:19.390 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:55:19.391 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:55:19.391 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:19.391 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:55:19.391 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T05:55:19.391 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:55:19.391 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T05:55:19.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-02-05T05:55:19.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:55:19.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:55:19.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:55:19.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:55:19.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:55:19.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:55:20.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:55:20.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:55:20.380 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:55:20.380 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:55:20.380 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:20.380 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:55:20.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:55:20.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:55:20.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:55:20.418 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-02-05T05:55:20.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-02-05T05:55:20.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-02-05T05:55:20.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:55:20.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:55:20.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:55:20.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:55:20.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:55:20.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:55:20.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:55:20.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:55:20.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:55:20.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:55:20.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:55:20.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:55:20.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:55:20.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:55:20.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:55:20.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:55:20.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:55:20.422 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:55:20.423 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:20.423 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:55:20.423 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:55:20.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:55:20.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:55:20.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:55:20.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:55:20.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:55:20.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:55:20.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:55:20.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:55:20.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:55:20.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:55:20.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:55:20.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:55:20.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:55:20.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:55:20.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d71c6a8d-fd48-4bfc-a494-1d0bf5575321 2024-02-05T05:55:20.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d71c6a8d-fd48-4bfc-a494-1d0bf5575321' 2024-02-05T05:55:20.425 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 d71c6a8d-fd48-4bfc-a494-1d0bf5575321 2024-02-05T05:55:20.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:55:20.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIeMBlTzZ/GxAA+mnqPaWcSnmvGCFSSh8C0w== 2024-02-05T05:55:20.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIeMBlTzZ/GxAA+mnqPaWcSnmvGCFSSh8C0w=="}' 2024-02-05T05:55:20.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d71c6a8d-fd48-4bfc-a494-1d0bf5575321 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:55:20.708 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:55:20.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:55:20.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIeMBlTzZ/GxAA+mnqPaWcSnmvGCFSSh8C0w== --osd-uuid d71c6a8d-fd48-4bfc-a494-1d0bf5575321 2024-02-05T05:55:20.764 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:20.780+0000 7f4451003640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:20.774 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:20.792+0000 7f4451003640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:20.776 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:20.792+0000 7f4451003640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:20.777 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:20.792+0000 7f4451003640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:55:20.777 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:20.792+0000 7f4451003640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:55:20.777 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:20.792+0000 7f4451003640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:55:20.875 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:20.892+0000 7f4451003640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:55:27.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:55:27.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:55:27.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:55:27.350 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T05:55:27.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:55:27.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:55:27.717 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T05:55:27.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:55:27.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:55:27.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:55:27.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:55:27.748 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:27.764+0000 7f208b3c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:27.750 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:27.768+0000 7f208b3c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:27.751 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:27.768+0000 7f208b3c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:28.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:55:28.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:55:28.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:55:28.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:55:28.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:55:28.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:28.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:55:28.050 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:55:28.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:28.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:28.312 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:28.328+0000 7f208b3c4640 -1 Falling back to public interface 2024-02-05T05:55:28.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:29.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:29.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:29.380 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:55:29.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:55:29.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:29.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:29.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:30.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:30.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:30.684 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:55:30.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:55:30.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:30.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:30.923 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:30.940+0000 7f208b3c4640 -1 osd.0 0 log_to_monitors true 2024-02-05T05:55:30.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:31.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:31.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:31.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:55:31.989 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:55:31.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:31.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:32.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:33.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:33.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:33.292 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:55:33.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:55:33.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:33.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:33.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:34.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:34.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:34.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:55:34.600 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:55:34.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:34.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:35.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:36.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:36.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:36.008 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:55:36.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:55:36.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:36.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:36.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:37.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:37.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:37.323 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:55:37.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:55:37.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:37.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:37.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:38.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:38.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:38.644 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:55:38.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:55:38.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:38.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:39.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:40.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:40.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:40.041 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:55:40.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:55:40.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:40.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:40.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:41.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:41.350 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:55:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:55:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:41.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:41.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:42.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:42.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:42.818 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:55:42.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:55:42.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:42.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:43.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:44.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:44.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:44.126 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:55:44.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:55:44.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:44.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:44.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:45.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:45.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:45.444 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:55:45.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:55:45.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:45.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:45.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:46.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:46.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:46.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:55:46.760 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:55:46.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:46.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:47.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:48.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:48.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:48.214 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:55:48.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:55:48.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:48.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:49.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:49.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:49.780 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:55:49.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:55:49.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:49.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:50.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:51.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:51.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:51.603 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:55:51.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:55:51.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:51.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:52.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:53.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:53.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:53.346 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:55:53.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:55:53.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:53.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:54.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:55.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:55.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:55.195 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:55:55.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:55:55.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:55.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:55.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:56.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:56.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:56.556 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:55:56.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:55:56.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:56.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:56.874 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/795233403,v1:127.0.0.1:6803/795233403] [v2:127.0.0.1:6804/795233403,v1:127.0.0.1:6805/795233403] exists,up d71c6a8d-fd48-4bfc-a494-1d0bf5575321 2024-02-05T05:55:56.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:55:56.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:55:56.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:55:56.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-02-05T05:55:56.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-02-05T05:55:56.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:55:56.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:55:56.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:55:56.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:55:56.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:55:56.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:55:56.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:55:56.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:55:56.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:55:56.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:55:56.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:55:56.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:55:56.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:55:56.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:55:56.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:55:56.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:55:56.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:55:56.878 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:55:56.878 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:56.878 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:55:56.880 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:55:56.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:55:56.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:55:56.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:55:56.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:55:56.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:55:56.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:55:56.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:55:56.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:55:56.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:55:56.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:55:56.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:55:56.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:55:56.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:55:56.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:55:56.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8ad5a377-74d2-4c8f-a335-9bbac82742ad 2024-02-05T05:55:56.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8ad5a377-74d2-4c8f-a335-9bbac82742ad' 2024-02-05T05:55:56.881 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 8ad5a377-74d2-4c8f-a335-9bbac82742ad 2024-02-05T05:55:56.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:55:56.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBseMBl6ub/NhAAAItooX0J+a9mM9vxoMtN/w== 2024-02-05T05:55:56.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBseMBl6ub/NhAAAItooX0J+a9mM9vxoMtN/w=="}' 2024-02-05T05:55:56.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8ad5a377-74d2-4c8f-a335-9bbac82742ad -i td/osd-scrub-repair/1/new.json 2024-02-05T05:55:57.261 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:55:57.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:55:57.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBseMBl6ub/NhAAAItooX0J+a9mM9vxoMtN/w== --osd-uuid 8ad5a377-74d2-4c8f-a335-9bbac82742ad 2024-02-05T05:55:57.313 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:57.331+0000 7fcc8c5fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:57.371 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:57.387+0000 7fcc8c5fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:57.373 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:57.391+0000 7fcc8c5fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:57.374 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:57.391+0000 7fcc8c5fd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:55:57.374 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:57.391+0000 7fcc8c5fd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:55:57.374 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:57.391+0000 7fcc8c5fd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:55:57.447 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:55:57.467+0000 7fcc8c5fd640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:56:04.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:56:04.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:56:04.231 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T05:56:04.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:56:04.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:56:04.601 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T05:56:04.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:56:04.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:56:04.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:56:04.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:56:04.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:56:04.632 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:04.647+0000 7f2aca90f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:04.634 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:04.651+0000 7f2aca90f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:04.636 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:04.651+0000 7f2aca90f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:04.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:56:04.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:56:04.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:56:04.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:56:04.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:56:04.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:04.936 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:56:04.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:56:04.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:04.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:05.206 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:05.223+0000 7f2aca90f640 -1 Falling back to public interface 2024-02-05T05:56:05.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:06.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:06.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:06.321 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:56:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:56:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:06.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:06.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:07.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:07.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:07.622 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:56:07.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:56:07.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:07.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:07.663 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:07.679+0000 7f2aca90f640 -1 osd.1 0 log_to_monitors true 2024-02-05T05:56:07.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:08.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:08.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:08.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:56:08.967 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:56:08.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:08.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:09.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:10.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:10.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:10.285 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:56:10.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:56:10.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:10.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:10.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:11.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:11.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:11.790 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:56:11.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:56:11.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:11.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:12.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:13.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:13.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:13.203 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:56:13.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:56:13.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:13.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:13.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:14.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:14.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:14.601 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:56:14.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:56:14.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:14.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:15.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:16.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:16.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:16.022 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:56:16.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:56:16.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:16.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:16.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:17.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:17.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:17.365 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:56:17.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:56:17.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:17.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:17.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:18.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:18.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:18.845 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:56:18.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:56:18.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:18.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:19.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:20.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:20.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:20.193 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:56:20.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:56:20.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:20.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:20.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:21.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:21.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:21.566 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:56:21.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:56:21.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:21.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:21.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:22.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:22.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:22.915 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:56:22.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:56:22.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:22.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:23.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:24.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:24.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:24.802 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:56:24.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:56:24.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:24.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:25.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:26.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:26.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:26.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:56:26.741 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:56:26.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:26.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:27.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:28.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:28.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:28.596 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:56:28.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:56:28.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:28.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:29.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:30.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:30.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:30.302 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:56:30.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:56:30.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:30.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:30.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:31.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:31.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:31.930 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:56:31.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:56:31.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:31.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:32.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:33.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:33.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:33.614 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:56:33.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:56:33.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:33.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:33.946 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1171712669,v1:127.0.0.1:6811/1171712669] [v2:127.0.0.1:6812/1171712669,v1:127.0.0.1:6813/1171712669] exists,up 8ad5a377-74d2-4c8f-a335-9bbac82742ad 2024-02-05T05:56:33.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:56:33.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:56:33.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:56:33.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-02-05T05:56:33.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-02-05T05:56:33.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:56:33.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:56:33.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:56:33.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:56:33.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:56:33.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:56:33.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:56:33.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:56:33.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:56:33.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:56:33.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:56:33.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:56:33.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:56:33.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:56:33.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:56:33.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:56:33.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:56:33.951 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:56:33.951 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:56:33.951 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:56:33.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:56:33.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:56:33.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:56:33.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:56:33.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:56:33.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:56:33.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:56:33.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:56:33.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:56:33.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:56:33.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:56:33.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:56:33.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:56:33.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:56:33.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:56:33.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b59305a1-b7cb-4629-8e6d-1493328eebd5 2024-02-05T05:56:33.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b59305a1-b7cb-4629-8e6d-1493328eebd5' 2024-02-05T05:56:33.954 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 b59305a1-b7cb-4629-8e6d-1493328eebd5 2024-02-05T05:56:33.955 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:56:33.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCReMBl8EZZOxAAsxh1mPSGtwhidW6uD1YP2Q== 2024-02-05T05:56:33.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCReMBl8EZZOxAAsxh1mPSGtwhidW6uD1YP2Q=="}' 2024-02-05T05:56:33.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b59305a1-b7cb-4629-8e6d-1493328eebd5 -i td/osd-scrub-repair/2/new.json 2024-02-05T05:56:34.508 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:56:34.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:56:34.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCReMBl8EZZOxAAsxh1mPSGtwhidW6uD1YP2Q== --osd-uuid b59305a1-b7cb-4629-8e6d-1493328eebd5 2024-02-05T05:56:34.566 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:34.582+0000 7f8cdfae6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:34.610 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:34.626+0000 7f8cdfae6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:34.612 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:34.626+0000 7f8cdfae6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:34.613 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:34.630+0000 7f8cdfae6640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:56:34.613 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:34.630+0000 7f8cdfae6640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:56:34.613 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:34.630+0000 7f8cdfae6640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:56:34.685 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:34.702+0000 7f8cdfae6640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:56:41.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:56:41.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:56:41.671 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T05:56:41.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:56:41.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:56:42.055 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T05:56:42.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:56:42.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:56:42.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:56:42.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:56:42.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:56:42.087 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:42.102+0000 7fc044d5a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:42.089 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:42.106+0000 7fc044d5a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:42.091 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:42.106+0000 7fc044d5a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:42.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:56:42.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:56:42.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:56:42.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:56:42.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:56:42.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:42.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:56:42.385 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T05:56:42.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:56:42.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:42.639 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:42.654+0000 7fc044d5a640 -1 Falling back to public interface 2024-02-05T05:56:42.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:43.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:43.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:43.713 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T05:56:43.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:56:43.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:43.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:56:44.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:45.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:45.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:45.017 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T05:56:45.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:56:45.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:45.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:56:45.271 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:56:45.286+0000 7fc044d5a640 -1 osd.2 0 log_to_monitors true 2024-02-05T05:56:45.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:46.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:46.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:46.321 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T05:56:46.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:56:46.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:46.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:56:46.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:47.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:47.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:47.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:56:47.667 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T05:56:47.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:47.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:56:48.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:49.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:49.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:49.041 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T05:56:49.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:56:49.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:49.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:56:49.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:50.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:50.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:50.594 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T05:56:50.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:56:50.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:50.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:56:50.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:51.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:51.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:51.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:56:51.976 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T05:56:51.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:51.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:56:52.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:53.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:53.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:53.325 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T05:56:53.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:56:53.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:53.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:56:53.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:54.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:54.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:54.643 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T05:56:54.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:56:54.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:54.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:56:54.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:55.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:55.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:55.975 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T05:56:55.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:56:55.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:55.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:56:56.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:57.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:57.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:57.488 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T05:56:57.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:56:57.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:57.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:56:57.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:58.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:58.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:58.806 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T05:56:58.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:56:58.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:58.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:56:59.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:00.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:00.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:00.269 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T05:57:00.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:57:00.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:00.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:00.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:01.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:01.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:01.786 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T05:57:01.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:57:01.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:01.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:02.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:03.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:03.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:03.534 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T05:57:03.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:57:03.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:03.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:04.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:05.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:05.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:05.363 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T05:57:05.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:57:05.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:05.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:06.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:07.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:07.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:07.172 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T05:57:07.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:57:07.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:07.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:07.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:08.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:08.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:08.778 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T05:57:08.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:57:08.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:08.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:09.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:10.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:10.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:10.504 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T05:57:10.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:57:10.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:10.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:11.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:12.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:12.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:12.068 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T05:57:12.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:57:12.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:12.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:12.398 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1456536786,v1:127.0.0.1:6819/1456536786] [v2:127.0.0.1:6820/1456536786,v1:127.0.0.1:6821/1456536786] exists,up b59305a1-b7cb-4629-8e6d-1493328eebd5 2024-02-05T05:57:12.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:57:12.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:57:12.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:57:12.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-02-05T05:57:12.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:57:12.697 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T05:57:12.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:57:12.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:57:13.197 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T05:57:13.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:57:14.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:57:16.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-02-05T05:57:16.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-02-05T05:57:17.411 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'foo' created 2024-02-05T05:57:17.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:57:18.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-02-05T05:57:18.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:57:18.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:57:18.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-02-05T05:57:18.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:57:18.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-02-05T05:57:18.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:57:18.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:57:20.157 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'ecpool' created 2024-02-05T05:57:20.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:57:21.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-02-05T05:57:21.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:57:21.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:57:21.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:57:21.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:57:21.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:57:21.186 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:57:21.186 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:57:21.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:57:21.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:57:21.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:57:21.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:57:21.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:57:21.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:57:21.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:57:21.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:57:21.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:57:21.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:57:21.634 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:57:21.634 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:57:21.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:57:21.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:21.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:57:21.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-02-05T05:57:21.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-02-05T05:57:21.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-02-05T05:57:21.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:21.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:57:21.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-02-05T05:57:21.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-02-05T05:57:21.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672974' 2024-02-05T05:57:21.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:21.943 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:57:22.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-02-05T05:57:22.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-02-05T05:57:22.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672974 2-64424509444' 2024-02-05T05:57:22.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:22.086 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-02-05T05:57:22.086 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:22.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:57:22.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-02-05T05:57:22.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:22.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-02-05T05:57:22.089 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836504 2024-02-05T05:57:22.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-02-05T05:57:22.090 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:57:22.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836504 2024-02-05T05:57:22.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:57:23.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:57:23.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:57:23.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-02-05T05:57:23.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:23.716 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-02-05T05:57:23.716 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:23.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:57:23.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-02-05T05:57:23.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:23.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-02-05T05:57:23.720 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672974 2024-02-05T05:57:23.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-02-05T05:57:23.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:57:24.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-02-05T05:57:24.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:24.038 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-02-05T05:57:24.038 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:24.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:57:24.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-02-05T05:57:24.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:24.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-02-05T05:57:24.043 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509444 2024-02-05T05:57:24.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-02-05T05:57:24.044 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:57:24.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509444 2024-02-05T05:57:24.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:57:24.370 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:57:24.370 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:57:24.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:57:24.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:57:24.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:57:24.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:57:24.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:57:24.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:57:24.766 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:57:24.766 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:57:25.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:57:25.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:57:25.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:57:25.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:57:25.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:57:25.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:57:25.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:57:25.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:57:25.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-02-05T05:57:25.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:57:25.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:57:25.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:57:25.618 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:57:25.619 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:57:25.619 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:57:25.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:57:25.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:57:25.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:57:25.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:57:25.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:57:25.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:57:25.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:57:25.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:57:25.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:57:26.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:57:26.017 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:57:26.017 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:57:26.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:57:26.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:26.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:57:26.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-02-05T05:57:26.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-02-05T05:57:26.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-02-05T05:57:26.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:26.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:57:26.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-02-05T05:57:26.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-02-05T05:57:26.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672976' 2024-02-05T05:57:26.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:26.334 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:57:26.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-02-05T05:57:26.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-02-05T05:57:26.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672976 2-64424509446' 2024-02-05T05:57:26.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:26.478 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-02-05T05:57:26.478 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:26.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:57:26.480 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-02-05T05:57:26.480 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:26.481 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836505 2024-02-05T05:57:26.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-02-05T05:57:26.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-02-05T05:57:26.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:57:26.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-02-05T05:57:26.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:57:27.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:57:27.785 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:57:28.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836505 2024-02-05T05:57:28.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:28.105 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-02-05T05:57:28.105 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:28.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:57:28.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-02-05T05:57:28.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:28.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-02-05T05:57:28.110 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672976 2024-02-05T05:57:28.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-02-05T05:57:28.111 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:57:28.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-02-05T05:57:28.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:28.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-02-05T05:57:28.423 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:28.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:57:28.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-02-05T05:57:28.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:28.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-02-05T05:57:28.426 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509446 2024-02-05T05:57:28.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-02-05T05:57:28.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:57:28.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-02-05T05:57:28.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:57:28.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:57:28.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:57:29.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:57:29.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:57:29.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:57:29.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:57:29.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:57:29.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:57:29.151 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:57:29.151 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:57:29.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:57:29.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:57:29.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:57:29.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:57:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:57:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:57:29.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:57:29.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-02-05T05:57:29.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:57:29.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-02-05T05:57:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-02-05T05:57:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:57:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:57:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-02-05T05:57:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:57:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:57:29.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:57:30.306 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T05:57:30.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:57:30.828 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T05:57:30.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:57:30.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:57:30.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:31.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-02-05T05:57:31.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T05:57:31.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:57:31.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-02-05T05:57:31.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-02-05T05:57:31.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:57:31.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:57:31.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:57:31.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:57:31.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:57:31.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:57:31.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:57:31.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:57:31.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:57:31.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:57:31.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:57:31.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:57:31.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:57:31.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:57:31.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:57:31.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:57:32.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:57:32.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:57:32.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:57:32.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:57:32.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:57:32.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:57:32.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:57:32.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:57:34.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:57:34.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:57:34.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:57:34.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:57:34.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:57:34.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:57:34.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:57:34.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:57:34.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:57:34.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:57:34.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:57:34.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:57:34.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:57:34.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:57:34.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:57:34.728 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:57:34.728 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:57:34.728 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:57:34.728 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:57:34.729 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:57:34.729 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:57:34.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:57:34.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:57:34.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:57:34.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:57:34.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:57:34.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:57:34.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:57:34.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:57:34.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:57:34.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:57:34.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:57:34.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:57:34.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:57:34.730 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:57:34.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:57:34.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:57:34.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:57:34.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:57:34.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:57:34.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:57:34.764 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:57:34.785+0000 7fd3d6a28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:57:34.767 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:57:34.785+0000 7fd3d6a28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:57:34.768 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:57:34.789+0000 7fd3d6a28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:57:35.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:57:35.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:57:35.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:57:35.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:57:35.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:57:35.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:35.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:57:35.044 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:57:35.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:35.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:57:35.303 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:57:35.321+0000 7fd3d6a28640 -1 Falling back to public interface 2024-02-05T05:57:35.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:36.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:36.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:36.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:57:36.384 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:57:36.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:36.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:57:36.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:37.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:37.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:37.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:57:37.692 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:57:37.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:37.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:57:37.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:38.242 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:57:38.261+0000 7fd3d6a28640 -1 osd.1 34 log_to_monitors true 2024-02-05T05:57:38.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:38.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:38.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:57:38.999 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:57:38.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:39.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:57:39.602 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 35 last_clean_interval [10,34) [v2:127.0.0.1:6810/3383287204,v1:127.0.0.1:6811/3383287204] [v2:127.0.0.1:6812/3383287204,v1:127.0.0.1:6813/3383287204] exists,up 8ad5a377-74d2-4c8f-a335-9bbac82742ad 2024-02-05T05:57:39.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:57:39.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:57:39.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:57:39.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:57:39.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:57:39.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:57:39.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:57:39.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:57:39.603 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:57:39.603 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:57:39.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:57:39.604 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:57:39.604 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:57:39.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:57:39.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:57:39.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:57:39.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:57:39.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:57:39.722 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:57:40.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:57:40.167 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:57:40.167 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:57:40.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:57:40.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:40.167 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:57:40.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-02-05T05:57:40.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-02-05T05:57:40.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-02-05T05:57:40.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:40.316 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:57:40.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-02-05T05:57:40.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-02-05T05:57:40.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-167503724546' 2024-02-05T05:57:40.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:40.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:57:40.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-02-05T05:57:40.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-02-05T05:57:40.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-167503724546 2-64424509450' 2024-02-05T05:57:40.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:40.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-02-05T05:57:40.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:40.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:57:40.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-02-05T05:57:40.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:40.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-02-05T05:57:40.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-02-05T05:57:40.640 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836509 2024-02-05T05:57:40.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:57:40.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-02-05T05:57:40.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:57:41.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:57:41.968 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:57:42.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-02-05T05:57:42.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:42.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724546 2024-02-05T05:57:42.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:57:42.290 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724546 2024-02-05T05:57:42.290 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:42.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-02-05T05:57:42.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2024-02-05T05:57:42.291 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 167503724546 2024-02-05T05:57:42.292 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:57:42.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-02-05T05:57:42.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:42.587 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-02-05T05:57:42.587 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:42.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:57:42.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-02-05T05:57:42.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:42.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-02-05T05:57:42.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-02-05T05:57:42.591 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509450 2024-02-05T05:57:42.591 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:57:42.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-02-05T05:57:42.906 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:57:42.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:57:42.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:57:43.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:57:43.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:57:43.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:57:43.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:57:43.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:57:43.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:57:43.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:57:43.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:57:43.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:57:43.634 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:57:43.634 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:57:43.634 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:57:44.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:57:44.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:57:44.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:57:44.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:57:44.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-02-05T05:57:44.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-02-05T05:57:44.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:57:44.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:57:44.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-02-05T05:57:44.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:57:44.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:57:44.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:57:44.463 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T05:57:44.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:57:45.116 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T05:57:45.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:57:45.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:57:45.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:45.347 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-02-05T05:57:45.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-02-05T05:57:45.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:57:45.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-02-05T05:57:45.350 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records in 2024-02-05T05:57:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records out 2024-02-05T05:57:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000165571 s, 12.4 MB/s 2024-02-05T05:57:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:57:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:57:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:57:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:57:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:57:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:57:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:57:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:57:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:57:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:57:45.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:57:45.352 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:57:45.352 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:57:45.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:57:45.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:57:45.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:57:46.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:57:46.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:57:46.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:57:46.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:57:46.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:57:46.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:57:46.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:57:46.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:57:48.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:57:48.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:57:48.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:57:48.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:57:48.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:57:48.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:57:48.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:57:48.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:57:48.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:57:48.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:57:48.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:57:48.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:57:48.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:57:48.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:57:48.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:57:48.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:57:48.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:57:48.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:57:48.737 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:57:48.737 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:57:48.737 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:57:48.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:57:48.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:57:48.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:57:48.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:57:48.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:57:48.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:57:48.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:57:48.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:57:48.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:57:48.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:57:48.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:57:48.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:57:48.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:57:48.739 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:57:48.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:57:48.739 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:57:48.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:57:48.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:57:48.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:57:48.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:57:48.779 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:57:48.796+0000 7f7f332b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:57:48.782 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:57:48.800+0000 7f7f332b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:57:48.784 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:57:48.804+0000 7f7f332b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:57:49.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:57:49.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:57:49.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:57:49.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:57:49.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:57:49.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:49.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:57:49.060 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:57:49.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:49.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:57:49.321 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:57:49.340+0000 7f7f332b1640 -1 Falling back to public interface 2024-02-05T05:57:49.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:50.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:50.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:50.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:57:50.418 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:57:50.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:50.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:57:50.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:51.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:51.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:51.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:57:51.716 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:57:51.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:51.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:57:52.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:52.241 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:57:52.260+0000 7f7f332b1640 -1 osd.0 43 log_to_monitors true 2024-02-05T05:57:53.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:53.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:53.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:57:53.028 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:57:53.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:53.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:57:53.352 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 48 up_thru 39 down_at 44 last_clean_interval [5,43) [v2:127.0.0.1:6802/108143036,v1:127.0.0.1:6803/108143036] [v2:127.0.0.1:6804/108143036,v1:127.0.0.1:6805/108143036] exists,up d71c6a8d-fd48-4bfc-a494-1d0bf5575321 2024-02-05T05:57:53.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:57:53.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:57:53.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:57:53.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:57:53.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:57:53.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:57:53.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:57:53.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:57:53.353 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:57:53.353 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:57:53.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:57:53.354 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:57:53.354 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:57:53.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:57:53.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:57:53.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:57:53.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:57:53.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:57:53.462 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:57:53.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:57:53.882 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:57:53.882 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:57:53.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:57:53.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:53.883 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:57:54.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430210 2024-02-05T05:57:54.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430210 2024-02-05T05:57:54.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210' 2024-02-05T05:57:54.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:54.107 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:57:54.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724550 2024-02-05T05:57:54.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724550 2024-02-05T05:57:54.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-167503724550' 2024-02-05T05:57:54.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:54.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:57:54.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-02-05T05:57:54.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-02-05T05:57:54.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-167503724550 2-64424509454' 2024-02-05T05:57:54.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:54.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:54.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430210 2024-02-05T05:57:54.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:57:54.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430210 2024-02-05T05:57:54.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:54.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430210 2024-02-05T05:57:54.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430210' 2024-02-05T05:57:54.533 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 206158430210 2024-02-05T05:57:54.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:57:54.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 206158430210 2024-02-05T05:57:54.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:57:55.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:57:55.835 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:57:56.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430210 -lt 206158430210 2024-02-05T05:57:56.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:56.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724550 2024-02-05T05:57:56.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:56.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:57:56.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724550 2024-02-05T05:57:56.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:56.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724550 2024-02-05T05:57:56.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724550' 2024-02-05T05:57:56.156 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 167503724550 2024-02-05T05:57:56.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:57:56.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724550 -lt 167503724550 2024-02-05T05:57:56.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:56.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:56.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-02-05T05:57:56.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:57:56.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-02-05T05:57:56.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:56.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-02-05T05:57:56.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-02-05T05:57:56.488 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509454 2024-02-05T05:57:56.489 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:57:56.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-02-05T05:57:56.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:57:56.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:57:56.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:57:57.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:57:57.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:57:57.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:57:57.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:57:57.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:57:57.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:57:57.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:57:57.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:57:57.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:57:57.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:57:57.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:57:57.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:57:57.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:57:57.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-02-05T05:57:57.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:57:57.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T05:57:57.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:57:57.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:57:58.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:57:58.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:57:58.086 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:57:58.086 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:57:58.087 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:57:58.087 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:57:58.087 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:57:58.087 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:57:58.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:57:58.399 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:57:58.399 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:57:58.399 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:57:58.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:57:58.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:57:58.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:57:58.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:57:58.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:57:58.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:57:58.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:57:58.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:57:58.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:57:59.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:57:59.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:57:59.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:57:59.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:57:59.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:57:59.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:57:59.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:57:59.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:57:59.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:57:59.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:57:59.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:57:59.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:57:59.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:57:59.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:57:59.716 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:57:59.716 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:57:59.716 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:00.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:58:00.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:58:00.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:58:00.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:58:00.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-02-05T05:58:00.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-02-05T05:58:00.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:58:00.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:58:00.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-02-05T05:58:00.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:58:00.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:58:00.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:58:00.528 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T05:58:00.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:58:01.005 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T05:58:01.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:58:01.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:58:01.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:58:01.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-02-05T05:58:01.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T05:58:01.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:58:01.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-02-05T05:58:01.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:58:01.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:58:01.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:58:01.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:58:01.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-02-05T05:58:01.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:58:01.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:58:01.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:58:01.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:58:01.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:58:01.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:58:01.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:58:01.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:58:01.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:58:01.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:58:01.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:58:01.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-02-05T05:58:01.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:01.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:01.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:58:01.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:01.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:58:01.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-02-05T05:58:03.838 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-02-05T05:58:04.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:58:04.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:58:04.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:58:04.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:58:04.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:58:04.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:58:04.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:58:04.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:58:04.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:58:04.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:58:04.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:58:04.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:58:04.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:58:04.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:58:04.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:58:04.377 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:58:04.377 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:58:04.377 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:58:04.377 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:58:04.377 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:58:04.377 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:58:04.377 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:58:04.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:58:04.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:58:04.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:58:04.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:58:04.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:58:04.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:58:04.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:58:04.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:58:04.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:58:04.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:58:04.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:58:04.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:58:04.379 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:58:04.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:58:04.380 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:58:04.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:58:04.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:58:04.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:58:04.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:58:04.415 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:04.432+0000 7fabd4a8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:04.416 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:04.436+0000 7fabd4a8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:04.418 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:04.436+0000 7fabd4a8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:04.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:58:04.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:58:04.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:58:04.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:58:04.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:58:04.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:04.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:58:04.702 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:58:04.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:58:04.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:04.954 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:04.972+0000 7fabd4a8b640 -1 Falling back to public interface 2024-02-05T05:58:05.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:06.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:06.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:06.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:58:06.055 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:58:06.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:06.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:58:06.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:07.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:07.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:07.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:58:07.366 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:58:07.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:07.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:58:07.629 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:07.648+0000 7fabd4a8b640 -1 osd.1 52 log_to_monitors true 2024-02-05T05:58:07.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:08.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:08.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:08.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:58:08.730 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:58:08.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:08.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:58:09.307 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [39,52) [v2:127.0.0.1:6810/548699311,v1:127.0.0.1:6811/548699311] [v2:127.0.0.1:6812/548699311,v1:127.0.0.1:6813/548699311] exists,up 8ad5a377-74d2-4c8f-a335-9bbac82742ad 2024-02-05T05:58:09.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:58:09.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:58:09.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:58:09.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:58:09.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:58:09.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:58:09.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:58:09.308 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:58:09.309 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:58:09.309 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:58:09.309 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:58:09.309 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:58:09.309 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:58:09.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:58:09.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:58:09.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:58:09.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:58:09.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:58:09.423 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:58:09.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:58:09.863 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:58:09.863 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:58:09.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:58:09.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:09.864 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:58:10.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430214 2024-02-05T05:58:10.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430214 2024-02-05T05:58:10.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214' 2024-02-05T05:58:10.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:10.035 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:58:10.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135874 2024-02-05T05:58:10.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135874 2024-02-05T05:58:10.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214 1-244813135874' 2024-02-05T05:58:10.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:10.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:58:10.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-02-05T05:58:10.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-02-05T05:58:10.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214 1-244813135874 2-64424509458' 2024-02-05T05:58:10.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:10.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:10.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430214 2024-02-05T05:58:10.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:58:10.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430214 2024-02-05T05:58:10.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:10.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430214 2024-02-05T05:58:10.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430214' 2024-02-05T05:58:10.323 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 206158430214 2024-02-05T05:58:10.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:58:10.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430213 -lt 206158430214 2024-02-05T05:58:10.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:58:11.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:58:11.631 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:58:11.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430214 -lt 206158430214 2024-02-05T05:58:11.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:11.940 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135874 2024-02-05T05:58:11.940 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:11.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:58:11.943 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135874 2024-02-05T05:58:11.944 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:11.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135874 2024-02-05T05:58:11.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-02-05T05:58:11.945 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 244813135874 2024-02-05T05:58:11.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:58:12.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135874 -lt 244813135874 2024-02-05T05:58:12.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:12.251 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:12.251 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-02-05T05:58:12.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:58:12.253 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-02-05T05:58:12.253 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:12.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-02-05T05:58:12.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-02-05T05:58:12.255 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509458 2024-02-05T05:58:12.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:58:12.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-02-05T05:58:12.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:58:12.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:12.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:58:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:58:13.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:58:13.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:58:13.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:58:13.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:58:13.024 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:58:13.024 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:58:13.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:58:13.332 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:58:13.332 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:13.332 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:13.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:58:13.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:58:13.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:58:13.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:58:13.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-02-05T05:58:13.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-02-05T05:58:13.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:58:13.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:58:13.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-02-05T05:58:13.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:58:13.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:58:13.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:58:14.136 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T05:58:14.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:58:14.714 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T05:58:14.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:58:14.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:58:14.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:58:15.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-02-05T05:58:15.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-02-05T05:58:15.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:58:15.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-02-05T05:58:15.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-02-05T05:58:15.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-02-05T05:58:15.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T05:58:15.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:58:15.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:58:15.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:58:15.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:58:15.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T05:58:15.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:58:15.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:58:15.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:58:15.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:58:15.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:58:15.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:58:15.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:58:15.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:58:15.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:58:15.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:58:16.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:58:16.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T05:58:16.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:16.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:16.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:58:16.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:16.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:58:16.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T05:58:18.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:58:18.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:58:18.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:58:18.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:58:18.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:58:18.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:58:18.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:58:18.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:58:18.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:58:18.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:58:18.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:58:18.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:58:18.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:58:18.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:58:18.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:58:18.764 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:58:18.764 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:58:18.764 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:58:18.765 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:58:18.765 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:58:18.766 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:58:18.766 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:58:18.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:58:18.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:58:18.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:58:18.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:58:18.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:58:18.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:58:18.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:58:18.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:58:18.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:58:18.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:58:18.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:58:18.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:58:18.767 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:58:18.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:58:18.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:58:18.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:58:18.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:58:18.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:58:18.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:58:18.800 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:18.820+0000 7fb500502640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:18.803 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:18.820+0000 7fb500502640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:18.804 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:18.824+0000 7fb500502640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:19.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:58:19.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:58:19.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:58:19.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:58:19.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:58:19.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:19.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:58:19.113 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:58:19.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:19.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:58:19.354 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:19.372+0000 7fb500502640 -1 Falling back to public interface 2024-02-05T05:58:19.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:20.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:20.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:20.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:58:20.453 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:58:20.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:20.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:58:20.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:21.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:21.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:21.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:58:21.770 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:58:21.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:21.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:58:22.024 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:22.044+0000 7fb500502640 -1 osd.0 61 log_to_monitors true 2024-02-05T05:58:22.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:23.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:23.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:23.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:58:23.085 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:58:23.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:23.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:58:23.633 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 66 up_thru 66 down_at 62 last_clean_interval [48,61) [v2:127.0.0.1:6802/3810659009,v1:127.0.0.1:6803/3810659009] [v2:127.0.0.1:6804/3810659009,v1:127.0.0.1:6805/3810659009] exists,up d71c6a8d-fd48-4bfc-a494-1d0bf5575321 2024-02-05T05:58:23.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:58:23.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:58:23.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:58:23.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:58:23.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:58:23.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:58:23.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:58:23.634 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:58:23.634 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:58:23.634 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:58:23.634 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:58:23.634 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:58:23.634 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:58:23.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:58:23.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:58:23.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:58:23.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:58:23.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:58:23.750 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:58:24.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:58:24.344 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:58:24.344 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:58:24.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:58:24.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:24.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:58:24.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-02-05T05:58:24.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-02-05T05:58:24.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538' 2024-02-05T05:58:24.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:24.596 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:58:24.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135878 2024-02-05T05:58:24.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135878 2024-02-05T05:58:24.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-244813135878' 2024-02-05T05:58:24.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:24.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:58:24.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-02-05T05:58:24.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-02-05T05:58:24.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-244813135878 2-64424509462' 2024-02-05T05:58:24.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:24.945 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841538 2024-02-05T05:58:24.945 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:24.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:58:24.947 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841538 2024-02-05T05:58:24.947 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:24.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-02-05T05:58:24.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841538' 2024-02-05T05:58:24.948 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 283467841538 2024-02-05T05:58:24.949 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:58:25.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841538 -lt 283467841538 2024-02-05T05:58:25.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:25.286 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135878 2024-02-05T05:58:25.286 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:25.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:58:25.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135878 2024-02-05T05:58:25.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:25.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135878 2024-02-05T05:58:25.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135878' 2024-02-05T05:58:25.291 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 244813135878 2024-02-05T05:58:25.291 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:58:25.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135878 -lt 244813135878 2024-02-05T05:58:25.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:25.618 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-02-05T05:58:25.618 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:25.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:58:25.621 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-02-05T05:58:25.621 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:25.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-02-05T05:58:25.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-02-05T05:58:25.622 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509462 2024-02-05T05:58:25.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:58:25.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509462 2024-02-05T05:58:25.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:58:26.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:58:26.919 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:58:27.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-02-05T05:58:27.221 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:58:27.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:27.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:27.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:58:27.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:58:27.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:58:27.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:58:27.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:58:27.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:58:27.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:58:27.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:58:27.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:58:27.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:58:28.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:28.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:28.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:58:28.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-02-05T05:58:28.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:58:28.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T05:58:28.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:58:28.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:58:28.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:58:28.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:58:28.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:58:28.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:58:28.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:58:28.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:58:28.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:58:28.461 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:58:28.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:58:28.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:58:28.788 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:28.788 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:29.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:58:29.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:58:29.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:58:29.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:58:29.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:58:29.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:58:29.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:58:29.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:58:29.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:58:29.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:58:29.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:58:29.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:58:29.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:58:29.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:58:29.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:58:29.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:58:29.757 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:58:29.757 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:58:29.757 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:58:29.757 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:58:29.757 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:58:29.758 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:58:30.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:58:30.050 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:58:30.050 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:30.050 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:30.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:58:30.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:58:30.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:58:30.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T05:58:30.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:58:30.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:58:30.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T05:58:30.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:58:30.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T05:58:30.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:58:30.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:58:30.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T05:58:30.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:58:30.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:58:30.401 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:58:30.401 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:58:30.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:58:30.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:58:30.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:58:30.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:58:30.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T05:58:30.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:30.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:30.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T05:58:30.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:30.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:58:30.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T05:58:32.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T05:58:32.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:58:32.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:58:32.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:58:32.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:58:32.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:58:32.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:58:32.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:58:32.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:58:32.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:58:32.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:58:32.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:58:32.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:58:32.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:58:32.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:58:32.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:58:32.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:58:32.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:58:32.997 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:58:32.997 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:58:32.998 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:58:32.998 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:58:32.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:58:32.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:58:32.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:58:32.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:58:32.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:58:32.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:58:32.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:58:32.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:58:32.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:58:32.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:58:32.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:58:32.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:58:32.999 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.2 2024-02-05T05:58:32.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:58:32.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:58:32.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:58:33.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:58:33.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:58:33.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:58:33.032 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:33.051+0000 7f020c34c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:33.068 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:33.087+0000 7f020c34c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:33.069 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:33.087+0000 7f020c34c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:33.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:58:33.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:58:33.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:58:33.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:58:33.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:58:33.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:33.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:58:33.321 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:58:33.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:33.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:58:33.613 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:33.631+0000 7f020c34c640 -1 Falling back to public interface 2024-02-05T05:58:33.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:34.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:34.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:34.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:58:34.664 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:58:34.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:34.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:58:34.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:35.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:35.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:35.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:58:35.971 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:58:35.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:35.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:58:36.245 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:36.263+0000 7f020c34c640 -1 osd.2 69 log_to_monitors true 2024-02-05T05:58:36.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:37.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:37.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:37.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:58:37.294 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:58:37.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:37.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:58:37.732 INFO:tasks.workunit.client.0.smithi148.stderr:osd.2 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/2385324036,v1:127.0.0.1:6819/2385324036] [v2:127.0.0.1:6820/2385324036,v1:127.0.0.1:6821/2385324036] exists,up b59305a1-b7cb-4629-8e6d-1493328eebd5 2024-02-05T05:58:37.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:58:37.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:58:37.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:58:37.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:58:37.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:58:37.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:58:37.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:58:37.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:58:37.733 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:58:37.733 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:58:37.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:58:37.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:58:37.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:58:37.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:58:37.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:58:37.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:58:37.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:58:37.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:58:37.833 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:58:38.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:58:38.237 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:58:38.237 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:58:38.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:58:38.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:38.238 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:58:38.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841542 2024-02-05T05:58:38.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841542 2024-02-05T05:58:38.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841542' 2024-02-05T05:58:38.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:38.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:58:38.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135882 2024-02-05T05:58:38.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135882 2024-02-05T05:58:38.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841542 1-244813135882' 2024-02-05T05:58:38.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:38.581 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:58:38.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-02-05T05:58:38.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-02-05T05:58:38.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841542 1-244813135882 2-313532612610' 2024-02-05T05:58:38.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:38.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841542 2024-02-05T05:58:38.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:38.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:58:38.727 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841542 2024-02-05T05:58:38.727 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:38.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841542 2024-02-05T05:58:38.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841542' 2024-02-05T05:58:38.728 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 283467841542 2024-02-05T05:58:38.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:58:39.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841542 -lt 283467841542 2024-02-05T05:58:39.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:39.032 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135882 2024-02-05T05:58:39.032 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:39.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:58:39.034 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135882 2024-02-05T05:58:39.034 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:39.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135882 2024-02-05T05:58:39.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135882' 2024-02-05T05:58:39.035 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 244813135882 2024-02-05T05:58:39.035 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:58:39.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135882 -lt 244813135882 2024-02-05T05:58:39.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:39.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612610 2024-02-05T05:58:39.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:39.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:58:39.343 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612610 2024-02-05T05:58:39.343 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:39.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-02-05T05:58:39.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-02-05T05:58:39.344 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 313532612610 2024-02-05T05:58:39.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:58:39.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612610 -lt 313532612610 2024-02-05T05:58:39.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:58:39.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:39.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:40.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:58:40.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:58:40.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:58:40.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:58:40.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:58:40.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:58:40.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:58:40.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:58:40.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:58:40.334 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:58:40.334 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:40.334 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:40.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:58:40.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:58:40.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:58:40.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-02-05T05:58:40.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T05:58:40.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:58:40.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:58:40.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T05:58:40.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:58:40.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T05:58:40.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:58:40.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:58:40.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T05:58:40.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:58:40.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:58:40.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:58:40.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:58:40.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:58:40.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:58:40.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:58:40.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:58:40.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T05:58:40.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:40.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:40.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T05:58:40.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:40.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:58:40.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T05:58:43.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T05:58:43.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:58:43.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:58:43.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:58:43.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:58:43.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:58:43.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:58:43.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:58:43.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:58:43.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:58:43.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:58:43.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:58:43.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:58:43.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:58:43.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:58:43.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:58:43.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:58:43.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:58:43.061 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:58:43.061 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:58:43.061 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:58:43.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:58:43.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:58:43.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:58:43.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:58:43.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:58:43.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:58:43.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:58:43.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:58:43.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:58:43.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:58:43.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:58:43.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:58:43.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:58:43.064 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.2 2024-02-05T05:58:43.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:58:43.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:58:43.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:58:43.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:58:43.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:58:43.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:58:43.098 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:43.115+0000 7fa4f634b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:43.135 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:43.155+0000 7fa4f634b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:43.137 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:43.155+0000 7fa4f634b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:43.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:58:43.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:58:43.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:58:43.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:58:43.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:58:43.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:43.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:58:43.387 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:58:43.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:43.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:58:43.673 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:43.691+0000 7fa4f634b640 -1 Falling back to public interface 2024-02-05T05:58:43.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:44.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:44.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:44.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:58:44.731 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:58:44.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:44.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:58:45.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:46.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:46.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:46.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:58:46.040 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:58:46.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:46.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:58:46.234 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:58:46.251+0000 7fa4f634b640 -1 osd.2 75 log_to_monitors true 2024-02-05T05:58:46.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:47.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:47.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:47.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:58:47.362 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:58:47.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:47.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:58:47.789 INFO:tasks.workunit.client.0.smithi148.stderr:osd.2 up in weight 1 up_from 79 up_thru 79 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/4283742444,v1:127.0.0.1:6819/4283742444] [v2:127.0.0.1:6820/4283742444,v1:127.0.0.1:6821/4283742444] exists,up b59305a1-b7cb-4629-8e6d-1493328eebd5 2024-02-05T05:58:47.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:58:47.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:58:47.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:58:47.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:58:47.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:58:47.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:58:47.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:58:47.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:58:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:58:47.792 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:58:47.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:58:47.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:58:47.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:58:47.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:58:47.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:58:47.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:58:47.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:58:47.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:58:47.908 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:58:48.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:58:48.256 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:58:48.256 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:58:48.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:58:48.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:48.256 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:58:48.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841545 2024-02-05T05:58:48.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841545 2024-02-05T05:58:48.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545' 2024-02-05T05:58:48.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:48.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:58:48.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135885 2024-02-05T05:58:48.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135885 2024-02-05T05:58:48.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545 1-244813135885' 2024-02-05T05:58:48.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:48.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:58:48.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416386 2024-02-05T05:58:48.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416386 2024-02-05T05:58:48.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545 1-244813135885 2-339302416386' 2024-02-05T05:58:48.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:48.711 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841545 2024-02-05T05:58:48.712 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:48.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:58:48.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841545 2024-02-05T05:58:48.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:48.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841545 2024-02-05T05:58:48.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841545' 2024-02-05T05:58:48.714 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 283467841545 2024-02-05T05:58:48.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:58:49.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841545 -lt 283467841545 2024-02-05T05:58:49.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:49.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135885 2024-02-05T05:58:49.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:49.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:58:49.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135885 2024-02-05T05:58:49.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:49.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135885 2024-02-05T05:58:49.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135885' 2024-02-05T05:58:49.019 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 244813135885 2024-02-05T05:58:49.019 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:58:49.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135885 -lt 244813135885 2024-02-05T05:58:49.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:49.367 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-339302416386 2024-02-05T05:58:49.367 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:49.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:58:49.370 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-339302416386 2024-02-05T05:58:49.370 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:49.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416386 2024-02-05T05:58:49.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 339302416386' 2024-02-05T05:58:49.371 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 339302416386 2024-02-05T05:58:49.372 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:58:49.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416386 -lt 339302416386 2024-02-05T05:58:49.691 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:58:49.691 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:49.691 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:50.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:58:50.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:58:50.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:58:50.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:58:50.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:58:50.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:58:50.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:58:50.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:58:50.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:58:50.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:58:50.431 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:50.431 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:50.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T05:58:50.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T05:58:50.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:58:50.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:58:50.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:58:50.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:58:50.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:58:50.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:58:50.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:58:50.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:58:50.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:58:50.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:58:50.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:58:50.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:58:51.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:58:51.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:58:51.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:51.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:51.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T05:58:51.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:58:51.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:58:51.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:58:51.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:58:51.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:58:51.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:58:51.552 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:58:51.552 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:58:51.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:58:51.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:58:51.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:58:51.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:58:51.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:58:52.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:58:52.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:58:52.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:58:52.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:58:52.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:58:52.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:58:52.120 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:58:52.120 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:58:52.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:58:52.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:58:52.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:52.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:52.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T05:58:52.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:58:52.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:58:52.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:58:52.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:58:52.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:58:52.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:58:52.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:58:52.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:58:53.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:58:53.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:58:53.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T05:58:53.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:58:53.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T05:58:53.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:58:53.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:58:53.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:58:53.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:58:53.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:58:53.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:58:53.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:58:53.541 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:58:53.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:58:53.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:58:53.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:53.859 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:54.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T05:58:54.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:58:54.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:58:54.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:58:54.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:58:54.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:58:54.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:58:54.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:58:54.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:58:54.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:58:54.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:58:54.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-02-05T05:58:54.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:58:54.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-02-05T05:58:55.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:58:55.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:58:55.418 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:58:55.418 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:58:55.418 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:58:55.418 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:58:55.419 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:58:55.419 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:58:55.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:58:55.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:58:55.732 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:55.732 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:56.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:58:56.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:58:56.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:58:56.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-02-05T05:58:56.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:58:56.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-02-05T05:58:56.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-02-05T05:58:56.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:58:56.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:58:56.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-02-05T05:58:56.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:58:56.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:58:56.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:58:56.482 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T05:58:56.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:58:57.051 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T05:58:57.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:58:57.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:58:57.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:58:57.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-02-05T05:58:57.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T05:58:57.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:58:57.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-02-05T05:58:57.470 INFO:tasks.workunit.client.0.smithi148.stderr:2+0 records in 2024-02-05T05:58:57.470 INFO:tasks.workunit.client.0.smithi148.stderr:2+0 records out 2024-02-05T05:58:57.470 INFO:tasks.workunit.client.0.smithi148.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000174346 s, 23.5 MB/s 2024-02-05T05:58:57.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:58:57.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:58:57.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:58:57.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:58:57.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:58:57.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:58:57.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:58:57.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:58:57.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:58:57.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:58:57.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:58:57.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:58:57.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:58:57.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:58:57.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:58:57.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:58:58.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:58:58.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:58:58.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:58.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:58.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:58:58.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:58.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:58:58.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:59:00.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:59:00.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:59:00.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:59:00.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:59:00.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:59:00.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:59:00.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:59:00.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:59:00.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:59:00.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:59:00.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:59:00.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:59:00.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:59:00.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:59:00.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:59:00.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:59:00.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:59:00.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:59:00.660 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:59:00.660 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:59:00.660 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:59:00.660 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:59:00.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:59:00.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:59:00.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:59:00.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:59:00.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:59:00.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:59:00.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:59:00.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:59:00.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:59:00.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:59:00.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:59:00.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:59:00.661 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:59:00.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:59:00.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:59:00.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:59:00.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:59:00.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:59:00.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:59:00.699 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:00.718+0000 7f524c703640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:00.702 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:00.718+0000 7f524c703640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:00.704 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:00.722+0000 7f524c703640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:00.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:59:00.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:59:00.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:59:00.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:59:00.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:59:00.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:00.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:59:00.982 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:59:00.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:00.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:59:01.237 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:01.254+0000 7f524c703640 -1 Falling back to public interface 2024-02-05T05:59:01.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:02.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:02.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:02.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:59:02.325 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:59:02.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:02.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:59:02.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:03.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:03.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:03.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:59:03.625 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:59:03.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:03.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:59:03.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:03.965 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:03.982+0000 7f524c703640 -1 osd.1 84 log_to_monitors true 2024-02-05T05:59:04.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:04.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:04.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:59:04.942 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:59:04.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:04.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:59:05.383 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [57,84) [v2:127.0.0.1:6810/3367927345,v1:127.0.0.1:6811/3367927345] [v2:127.0.0.1:6812/3367927345,v1:127.0.0.1:6813/3367927345] exists,up 8ad5a377-74d2-4c8f-a335-9bbac82742ad 2024-02-05T05:59:05.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:59:05.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:59:05.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:59:05.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:59:05.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:59:05.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:59:05.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:59:05.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:59:05.384 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:59:05.384 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:59:05.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:59:05.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:59:05.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:59:05.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:59:05.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:59:05.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:59:05.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:59:05.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:59:05.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:59:05.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:59:05.895 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:59:05.895 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:59:05.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:59:05.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:59:05.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:59:06.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841549 2024-02-05T05:59:06.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841549 2024-02-05T05:59:06.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841549' 2024-02-05T05:59:06.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:59:06.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:59:06.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-02-05T05:59:06.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-02-05T05:59:06.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841549 1-377957122050' 2024-02-05T05:59:06.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:59:06.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:59:06.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416391 2024-02-05T05:59:06.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416391 2024-02-05T05:59:06.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841549 1-377957122050 2-339302416391' 2024-02-05T05:59:06.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:59:06.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841549 2024-02-05T05:59:06.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:59:06.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:59:06.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841549 2024-02-05T05:59:06.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:59:06.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841549 2024-02-05T05:59:06.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841549' 2024-02-05T05:59:06.503 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 283467841549 2024-02-05T05:59:06.504 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:59:06.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841548 -lt 283467841549 2024-02-05T05:59:06.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:59:07.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:59:07.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:59:08.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841549 -lt 283467841549 2024-02-05T05:59:08.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:59:08.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122050 2024-02-05T05:59:08.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:59:08.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:59:08.137 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122050 2024-02-05T05:59:08.137 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:59:08.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-02-05T05:59:08.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2024-02-05T05:59:08.139 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 377957122050 2024-02-05T05:59:08.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:59:08.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122050 -lt 377957122050 2024-02-05T05:59:08.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:59:08.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-339302416391 2024-02-05T05:59:08.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:59:08.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:59:08.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:59:08.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-339302416391 2024-02-05T05:59:08.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416391 2024-02-05T05:59:08.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 339302416391' 2024-02-05T05:59:08.454 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 339302416391 2024-02-05T05:59:08.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:59:08.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416391 -lt 339302416391 2024-02-05T05:59:08.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:59:08.772 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:08.772 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:09.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:59:09.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:59:09.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:59:09.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:59:09.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:59:09.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:59:09.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:59:09.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:59:09.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:59:09.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:59:09.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:09.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:09.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:59:09.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:59:09.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:59:09.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:59:09.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-02-05T05:59:09.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-02-05T05:59:09.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:59:09.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:59:09.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-02-05T05:59:09.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:59:09.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:59:09.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:59:10.333 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T05:59:10.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:59:10.902 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T05:59:10.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:59:10.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:59:10.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:59:11.201 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-02-05T05:59:11.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-02-05T05:59:11.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:59:11.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-02-05T05:59:11.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:59:11.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:59:11.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:59:11.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:59:11.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-02-05T05:59:11.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:59:11.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:59:11.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:59:11.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:59:11.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:59:11.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:59:11.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:59:11.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:59:11.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:59:11.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:59:11.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:59:11.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-02-05T05:59:11.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:59:11.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:59:11.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:59:11.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:59:11.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:59:11.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-02-05T05:59:14.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:59:14.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:59:14.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:59:14.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:59:14.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:59:14.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:59:14.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:59:14.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:59:14.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:59:14.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:59:14.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:59:14.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:59:14.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:59:14.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:59:14.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:59:14.049 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:59:14.049 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:59:14.049 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:59:14.049 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:59:14.049 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:59:14.049 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:59:14.049 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:59:14.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:59:14.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:59:14.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:59:14.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:59:14.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:59:14.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:59:14.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:59:14.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:59:14.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:59:14.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:59:14.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:59:14.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:59:14.051 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:59:14.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:59:14.051 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:59:14.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:59:14.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:59:14.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:59:14.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:59:14.086 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:14.102+0000 7fb8fc7ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:14.089 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:14.106+0000 7fb8fc7ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:14.091 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:14.106+0000 7fb8fc7ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:14.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:59:14.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:59:14.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:59:14.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:59:14.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:59:14.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:14.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:59:14.371 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:59:14.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:14.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:14.628 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:14.642+0000 7fb8fc7ef640 -1 Falling back to public interface 2024-02-05T05:59:14.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:15.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:15.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:15.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:59:15.664 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:59:15.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:15.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:15.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:16.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:16.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:16.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:59:16.969 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:59:16.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:16.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:17.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:17.551 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:17.542+0000 7fb8fc7ef640 -1 osd.0 92 log_to_monitors true 2024-02-05T05:59:18.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:18.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:18.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:59:18.285 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:59:18.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:18.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:18.803 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 97 up_thru 88 down_at 93 last_clean_interval [66,92) [v2:127.0.0.1:6802/3325432605,v1:127.0.0.1:6803/3325432605] [v2:127.0.0.1:6804/3325432605,v1:127.0.0.1:6805/3325432605] exists,up d71c6a8d-fd48-4bfc-a494-1d0bf5575321 2024-02-05T05:59:18.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:59:18.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:59:18.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:59:18.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:59:18.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:59:18.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:59:18.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:59:18.804 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:59:18.804 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:59:18.804 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:59:18.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:59:18.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:59:18.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:59:18.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:59:18.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:59:18.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:59:18.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:59:18.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:59:18.927 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:59:19.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:59:19.435 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:59:19.435 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:59:19.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:59:19.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:59:19.435 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:59:19.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827714 2024-02-05T05:59:19.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827714 2024-02-05T05:59:19.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-416611827714' 2024-02-05T05:59:19.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:59:19.657 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:59:19.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122054 2024-02-05T05:59:19.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122054 2024-02-05T05:59:19.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-416611827714 1-377957122054' 2024-02-05T05:59:19.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:59:19.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:59:19.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416394 2024-02-05T05:59:19.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416394 2024-02-05T05:59:19.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-416611827714 1-377957122054 2-339302416394' 2024-02-05T05:59:19.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:59:19.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-416611827714 2024-02-05T05:59:19.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:59:19.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:59:19.947 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-416611827714 2024-02-05T05:59:19.947 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:59:19.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827714 2024-02-05T05:59:19.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 416611827714' 2024-02-05T05:59:19.949 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 416611827714 2024-02-05T05:59:19.950 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:59:20.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 416611827714 2024-02-05T05:59:20.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:59:21.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:59:21.258 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:59:21.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827714 -lt 416611827714 2024-02-05T05:59:21.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:59:21.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122054 2024-02-05T05:59:21.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:59:21.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:59:21.578 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122054 2024-02-05T05:59:21.578 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:59:21.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122054 2024-02-05T05:59:21.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122054' 2024-02-05T05:59:21.579 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 377957122054 2024-02-05T05:59:21.579 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:59:21.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122054 -lt 377957122054 2024-02-05T05:59:21.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:59:21.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-339302416394 2024-02-05T05:59:21.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:59:21.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:59:21.897 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-339302416394 2024-02-05T05:59:21.897 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:59:21.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416394 2024-02-05T05:59:21.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 339302416394' 2024-02-05T05:59:21.899 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 339302416394 2024-02-05T05:59:21.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:59:22.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416394 -lt 339302416394 2024-02-05T05:59:22.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:59:22.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:22.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:22.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:59:22.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:59:22.602 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:59:22.602 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:59:22.602 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:59:22.602 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:59:22.602 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:59:22.602 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:59:22.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:59:22.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:59:22.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:22.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:23.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T05:59:23.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T05:59:23.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:59:23.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:59:23.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:59:23.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:59:23.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:59:23.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:59:23.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:59:23.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:59:23.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:59:23.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:59:23.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:59:23.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:59:23.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:59:23.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:59:23.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:23.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:24.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T05:59:24.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:59:24.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:59:24.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:59:24.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:59:24.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:59:24.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:59:24.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:59:24.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:59:24.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:59:24.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:59:24.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:59:24.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:59:24.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:59:24.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:59:24.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:59:24.663 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:59:24.663 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:59:24.663 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:59:24.663 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:59:24.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:59:24.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:59:24.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:59:24.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:59:24.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:24.955 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:25.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:59:25.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:59:25.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:59:25.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-02-05T05:59:25.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T05:59:25.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:59:25.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:59:25.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:59:25.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:59:25.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T05:59:25.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:59:25.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:59:25.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:59:25.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:59:25.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:59:25.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:59:25.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:59:25.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:59:25.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:59:25.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:59:25.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:59:25.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T05:59:25.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:59:25.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:59:25.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:59:25.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:59:25.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:59:25.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T05:59:28.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:59:28.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:59:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:59:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:59:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:59:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:59:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:59:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:59:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:59:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:59:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:59:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:59:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:59:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:59:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:59:28.020 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:59:28.020 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:59:28.020 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:59:28.021 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:59:28.021 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:59:28.021 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:59:28.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:59:28.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:59:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:59:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:59:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:59:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:59:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:59:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:59:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:59:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:59:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:59:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:59:28.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:59:28.023 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T05:59:28.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:59:28.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:59:28.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:59:28.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:59:28.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:59:28.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:59:28.062 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:28.082+0000 7f9d19453640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:28.065 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:28.086+0000 7f9d19453640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:28.066 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:28.086+0000 7f9d19453640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:28.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:59:28.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:59:28.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:59:28.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:59:28.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:59:28.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:28.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:59:28.333 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:59:28.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:28.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:59:28.601 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:28.622+0000 7f9d19453640 -1 Falling back to public interface 2024-02-05T05:59:28.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:29.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:29.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:29.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:59:29.641 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:59:29.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:29.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:59:29.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:30.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:30.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:30.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:59:30.940 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:59:30.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:30.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:59:31.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:31.272 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:31.294+0000 7f9d19453640 -1 osd.1 99 log_to_monitors true 2024-02-05T05:59:32.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:32.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:32.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:59:32.252 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:59:32.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:32.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:59:32.732 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 104 up_thru 104 down_at 100 last_clean_interval [88,99) [v2:127.0.0.1:6810/1099036058,v1:127.0.0.1:6811/1099036058] [v2:127.0.0.1:6812/1099036058,v1:127.0.0.1:6813/1099036058] exists,up 8ad5a377-74d2-4c8f-a335-9bbac82742ad 2024-02-05T05:59:32.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:59:32.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:59:32.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:59:32.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:59:32.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:59:32.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:59:32.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:59:32.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:59:32.733 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:59:32.734 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:59:32.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:59:32.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:59:32.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:59:32.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:59:32.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:59:32.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:59:32.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:59:32.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:59:32.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:59:33.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:59:33.252 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:59:33.252 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:59:33.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:59:33.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:59:33.252 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:59:33.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827718 2024-02-05T05:59:33.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827718 2024-02-05T05:59:33.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-416611827718' 2024-02-05T05:59:33.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:59:33.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:59:33.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598786 2024-02-05T05:59:33.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598786 2024-02-05T05:59:33.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-416611827718 1-446676598786' 2024-02-05T05:59:33.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:59:33.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:59:33.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416398 2024-02-05T05:59:33.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416398 2024-02-05T05:59:33.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-416611827718 1-446676598786 2-339302416398' 2024-02-05T05:59:33.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:59:33.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-416611827718 2024-02-05T05:59:33.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:59:33.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:59:33.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-416611827718 2024-02-05T05:59:33.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:59:33.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827718 2024-02-05T05:59:33.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 416611827718' 2024-02-05T05:59:33.678 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 416611827718 2024-02-05T05:59:33.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:59:33.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827716 -lt 416611827718 2024-02-05T05:59:33.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:59:34.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:59:34.988 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:59:35.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827718 -lt 416611827718 2024-02-05T05:59:35.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:59:35.307 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-446676598786 2024-02-05T05:59:35.307 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:59:35.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:59:35.309 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-446676598786 2024-02-05T05:59:35.309 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:59:35.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598786 2024-02-05T05:59:35.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 446676598786' 2024-02-05T05:59:35.311 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 446676598786 2024-02-05T05:59:35.311 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:59:35.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598786 -lt 446676598786 2024-02-05T05:59:35.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:59:35.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:59:35.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-339302416398 2024-02-05T05:59:35.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:59:35.660 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:59:35.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-339302416398 2024-02-05T05:59:35.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416398 2024-02-05T05:59:35.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 339302416398' 2024-02-05T05:59:35.662 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 339302416398 2024-02-05T05:59:35.662 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:59:35.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416398 -lt 339302416398 2024-02-05T05:59:35.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:59:35.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:35.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:36.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:59:36.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:59:36.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:59:36.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:59:36.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:59:36.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:59:36.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:59:36.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:59:36.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:59:36.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:59:36.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:36.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:37.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:59:37.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:59:37.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:59:37.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:59:37.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-02-05T05:59:37.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-02-05T05:59:37.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:59:37.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:59:37.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-02-05T05:59:37.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:59:37.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:59:37.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:59:37.437 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T05:59:37.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:59:38.065 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T05:59:38.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:59:38.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:59:38.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:59:38.346 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-02-05T05:59:38.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T05:59:38.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:59:38.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-02-05T05:59:38.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-02-05T05:59:38.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-02-05T05:59:38.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-02-05T05:59:38.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:59:38.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:59:38.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:59:38.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:59:38.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-02-05T05:59:38.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:59:38.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:59:38.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:59:38.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:59:38.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:59:38.717 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:59:38.717 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:59:38.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:59:38.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:59:38.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:59:40.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:59:40.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-02-05T05:59:40.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:59:40.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:59:40.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:59:40.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:59:40.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:59:40.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-02-05T05:59:41.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:59:41.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:59:41.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:59:41.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:59:41.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:59:41.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:59:41.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:59:41.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:59:41.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:59:41.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:59:41.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:59:41.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:59:41.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:59:41.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:59:41.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:59:41.836 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:59:41.836 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:59:41.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:59:41.837 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:59:41.837 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:59:41.837 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:59:41.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:59:41.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:59:41.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:59:41.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:59:41.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:59:41.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:59:41.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:59:41.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:59:41.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:59:41.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:59:41.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:59:41.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:59:41.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:59:41.838 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:59:41.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:59:41.838 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:59:41.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:59:41.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:59:41.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:59:41.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:59:41.872 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:41.893+0000 7f40acd2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:41.875 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:41.897+0000 7f40acd2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:41.877 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:41.897+0000 7f40acd2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:42.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:59:42.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:59:42.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:59:42.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:59:42.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:59:42.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:42.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:59:42.168 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:59:42.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:42.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:42.418 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:42.437+0000 7f40acd2e640 -1 Falling back to public interface 2024-02-05T05:59:42.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:43.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:43.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:43.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:59:43.480 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:59:43.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:43.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:43.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:44.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:44.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:44.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:59:44.792 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T05:59:44.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:44.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:45.052 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:45.073+0000 7f40acd2e640 -1 osd.0 108 log_to_monitors true 2024-02-05T05:59:45.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:46.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:46.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:46.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:59:46.115 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T05:59:46.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:46.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:46.440 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 113 up_thru 113 down_at 109 last_clean_interval [97,108) [v2:127.0.0.1:6802/2175132637,v1:127.0.0.1:6803/2175132637] [v2:127.0.0.1:6804/2175132637,v1:127.0.0.1:6805/2175132637] exists,up d71c6a8d-fd48-4bfc-a494-1d0bf5575321 2024-02-05T05:59:46.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:59:46.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:59:46.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:59:46.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:59:46.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:59:46.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:59:46.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:59:46.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:59:46.441 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:59:46.441 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:59:46.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:59:46.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:59:46.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:59:46.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:59:46.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:59:46.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:59:46.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:59:46.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:59:46.549 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:59:46.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:59:46.871 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:59:46.871 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T05:59:46.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:59:46.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:59:46.872 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:59:47.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304450 2024-02-05T05:59:47.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304450 2024-02-05T05:59:47.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-485331304450' 2024-02-05T05:59:47.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:59:47.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:59:47.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598790 2024-02-05T05:59:47.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598790 2024-02-05T05:59:47.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-485331304450 1-446676598790' 2024-02-05T05:59:47.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:59:47.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:59:47.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416402 2024-02-05T05:59:47.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416402 2024-02-05T05:59:47.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-485331304450 1-446676598790 2-339302416402' 2024-02-05T05:59:47.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:59:47.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-485331304450 2024-02-05T05:59:47.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:59:47.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:59:47.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-485331304450 2024-02-05T05:59:47.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:59:47.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304450 2024-02-05T05:59:47.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 485331304450' 2024-02-05T05:59:47.641 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 485331304450 2024-02-05T05:59:47.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:59:47.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 485331304450 2024-02-05T05:59:47.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:59:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:59:48.975 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:59:49.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304450 -lt 485331304450 2024-02-05T05:59:49.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:59:49.297 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-446676598790 2024-02-05T05:59:49.297 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:59:49.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:59:49.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-446676598790 2024-02-05T05:59:49.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:59:49.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598790 2024-02-05T05:59:49.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 446676598790' 2024-02-05T05:59:49.301 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 446676598790 2024-02-05T05:59:49.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:59:49.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598790 -lt 446676598790 2024-02-05T05:59:49.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:59:49.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-339302416402 2024-02-05T05:59:49.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:59:49.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:59:49.612 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-339302416402 2024-02-05T05:59:49.612 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:59:49.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416402 2024-02-05T05:59:49.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 339302416402' 2024-02-05T05:59:49.613 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 339302416402 2024-02-05T05:59:49.614 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:59:49.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416402 -lt 339302416402 2024-02-05T05:59:49.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:59:49.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:49.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:50.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:59:50.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:59:50.311 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:59:50.311 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:59:50.311 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:59:50.311 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:59:50.311 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:59:50.312 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:59:50.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:59:50.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:59:50.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:50.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:50.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:59:50.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-02-05T05:59:50.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:59:50.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T05:59:50.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:59:50.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:59:51.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:59:51.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:59:51.051 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:59:51.051 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:59:51.051 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:59:51.051 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:59:51.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:59:51.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:59:51.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:59:51.343 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:59:51.344 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:51.344 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:51.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:59:51.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:59:51.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:59:51.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:59:51.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:59:51.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:59:51.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:59:51.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:59:51.696 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:59:52.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:59:52.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:59:52.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:59:52.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:59:52.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:59:52.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:59:52.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:59:52.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:59:52.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:59:52.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:59:52.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:59:52.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:59:52.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:59:52.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:59:52.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:59:52.555 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:52.555 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:52.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:59:52.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:59:52.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:59:52.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:59:52.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:59:52.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:59:52.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:59:52.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:59:52.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:59:53.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:59:53.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:59:53.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T05:59:53.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:59:53.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T05:59:53.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:59:53.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:59:53.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:59:53.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:59:53.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:59:53.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:59:53.783 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:59:53.783 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:59:54.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:59:54.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:59:54.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:54.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:54.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:59:54.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:59:54.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:59:54.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T05:59:54.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:59:54.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:59:54.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:59:54.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:59:54.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T05:59:54.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:59:54.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:59:54.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:59:54.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:59:54.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:59:54.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:59:54.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:59:54.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:59:54.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:59:54.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:59:55.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:59:55.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T05:59:55.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:59:55.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:59:55.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:59:55.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:59:55.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:59:55.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T05:59:58.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:59:58.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:59:58.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:59:58.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:59:58.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:59:58.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:59:58.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:59:58.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:59:58.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:59:58.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:59:58.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:59:58.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:59:58.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:59:58.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:59:58.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:59:58.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:59:58.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:59:58.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:59:58.105 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:59:58.106 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:59:58.106 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T05:59:58.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:59:58.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T05:59:58.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:59:58.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:59:58.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:59:58.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:59:58.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:59:58.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:59:58.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:59:58.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:59:58.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:59:58.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:59:58.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:59:58.107 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T05:59:58.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:59:58.107 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:59:58.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:59:58.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:59:58.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:59:58.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:59:58.147 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:58.169+0000 7fae01e31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:58.150 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:58.169+0000 7fae01e31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:58.152 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:58.173+0000 7fae01e31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:59:58.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:59:58.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:59:58.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:59:58.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:59:58.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:59:58.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:58.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:59:58.444 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T05:59:58.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:58.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:58.685 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T05:59:58.705+0000 7fae01e31640 -1 Falling back to public interface 2024-02-05T05:59:58.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:59.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:59.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:59.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:59:59.777 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T05:59:59.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:59.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:00.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:01.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:01.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:01.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:00:01.091 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:00:01.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:01.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:01.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:01.515 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:01.537+0000 7fae01e31640 -1 osd.0 115 log_to_monitors true 2024-02-05T06:00:02.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:02.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:02.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:00:02.402 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:00:02.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:02.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:02.721 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 120 up_thru 113 down_at 116 last_clean_interval [113,115) [v2:127.0.0.1:6802/3938869871,v1:127.0.0.1:6803/3938869871] [v2:127.0.0.1:6804/3938869871,v1:127.0.0.1:6805/3938869871] exists,up d71c6a8d-fd48-4bfc-a494-1d0bf5575321 2024-02-05T06:00:02.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:00:02.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:00:02.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:00:02.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:00:02.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:00:02.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:00:02.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:00:02.722 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:00:02.722 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:00:02.722 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:00:02.723 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:00:02.723 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:00:02.723 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:00:02.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:00:02.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:00:02.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:00:02.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:00:02.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:00:02.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:00:03.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:00:03.418 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:00:03.418 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:00:03.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:00:03.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:00:03.419 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:00:03.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075522 2024-02-05T06:00:03.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075522 2024-02-05T06:00:03.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-515396075522' 2024-02-05T06:00:03.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:00:03.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:00:03.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598794 2024-02-05T06:00:03.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598794 2024-02-05T06:00:03.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-515396075522 1-446676598794' 2024-02-05T06:00:03.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:00:03.955 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:00:04.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416406 2024-02-05T06:00:04.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416406 2024-02-05T06:00:04.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-515396075522 1-446676598794 2-339302416406' 2024-02-05T06:00:04.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:00:04.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-515396075522 2024-02-05T06:00:04.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:00:04.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:00:04.099 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-515396075522 2024-02-05T06:00:04.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:00:04.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075522 2024-02-05T06:00:04.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 515396075522' 2024-02-05T06:00:04.101 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 515396075522 2024-02-05T06:00:04.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:00:04.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 515396075522 2024-02-05T06:00:04.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:00:05.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:00:05.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:00:05.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075522 -lt 515396075522 2024-02-05T06:00:05.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:00:05.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-446676598794 2024-02-05T06:00:05.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:00:05.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:00:05.758 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-446676598794 2024-02-05T06:00:05.758 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:00:05.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598794 2024-02-05T06:00:05.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 446676598794' 2024-02-05T06:00:05.759 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 446676598794 2024-02-05T06:00:05.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:00:06.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598794 -lt 446676598794 2024-02-05T06:00:06.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:00:06.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-339302416406 2024-02-05T06:00:06.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:00:06.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:00:06.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-339302416406 2024-02-05T06:00:06.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:00:06.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416406 2024-02-05T06:00:06.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 339302416406' 2024-02-05T06:00:06.116 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 339302416406 2024-02-05T06:00:06.116 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:00:06.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416406 -lt 339302416406 2024-02-05T06:00:06.464 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:00:06.464 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:00:06.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:00:06.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:00:06.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:00:06.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:00:06.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:00:06.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:00:06.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:00:06.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:00:06.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:00:07.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T06:00:07.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:00:07.137 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:00:07.137 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:00:07.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-02-05T06:00:07.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-02-05T06:00:07.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:00:07.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-02-05T06:00:07.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:00:07.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:00:07.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:00:07.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:00:07.591 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:00:07.591 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:00:07.591 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:00:07.591 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:00:07.592 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:00:07.592 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:00:07.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T06:00:07.884 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:00:07.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:00:07.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:00:08.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-02-05T06:00:08.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-02-05T06:00:08.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:00:08.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:00:08.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:00:08.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:00:08.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:00:08.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:00:08.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:00:08.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:00:08.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:00:08.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:00:08.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:00:08.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:00:08.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:00:08.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:00:08.809 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:00:08.809 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:00:08.809 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:00:08.809 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:00:08.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:00:08.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:00:09.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:00:09.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:00:09.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:00:09.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:00:09.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:00:09.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:00:09.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:00:09.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-02-05T06:00:09.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-02-05T06:00:09.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T06:00:09.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-02-05T06:00:09.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-02-05T06:00:09.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:00:09.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-02-05T06:00:09.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-02-05T06:00:09.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-02-05T06:00:09.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-02-05T06:00:09.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:00:09.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:00:09.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:00:09.838 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:00:10.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-02-05T05:57:19.785662+0000 2024-02-05T06:00:10.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-02-05T06:00:10.454 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-02-05T06:00:10.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-02-05T05:57:19.785662+0000 2024-02-05T06:00:10.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T06:00:10.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:57:19.785662+0000 2024-02-05T06:00:10.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:00:10.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:00:10.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:00:10.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:00:10.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:00:10.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:00:10.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:00:10.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:00:10.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:57:19.785662+0000 '>' 2024-02-05T05:57:19.785662+0000 2024-02-05T06:00:10.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:00:11.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:00:11.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:00:11.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:00:11.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:00:11.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:00:11.783 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:00:11.783 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:00:12.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:57:19.785662+0000 '>' 2024-02-05T05:57:19.785662+0000 2024-02-05T06:00:12.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:00:13.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:00:13.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:00:13.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:00:13.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:00:13.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:00:13.102 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:00:13.102 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:00:13.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:00:10.572271+0000 '>' 2024-02-05T05:57:19.785662+0000 2024-02-05T06:00:13.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:00:13.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-02-05T06:00:13.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-02-05T06:00:13.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-02-05T06:00:13.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-02-05T06:00:13.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-02-05T06:00:13.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-02-05T06:00:13.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-02-05T06:00:13.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=120 2024-02-05T06:00:13.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T06:00:13.564 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T06:00:13.564 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T06:00:13.565 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T06:00:13.565 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T06:00:13.565 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T06:00:13.565 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T06:00:13.565 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T06:00:13.565 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T06:00:13.565 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T06:00:13.565 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T06:00:13.565 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T06:00:13.565 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T06:00:13.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T06:00:13.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T06:00:13.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T06:00:13.610 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T06:00:13.610 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T06:00:13.610 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T06:00:13.610 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T06:00:13.610 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T06:00:13.610 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T06:00:13.610 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T06:00:13.610 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T06:00:13.610 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T06:00:13.610 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T06:00:13.610 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T06:00:13.611 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-02-05T06:00:13.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T06:00:13.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T06:00:13.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T06:00:13.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T06:00:13.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-02-05T06:00:13.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-02-05T06:00:13.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-02-05T06:00:13.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-02-05T06:00:13.656 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:00:13.656 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:00:13.656 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:00:13.657 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:00:13.657 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:00:13.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-02-05T05:57:19.785662+0000 2024-02-05T06:00:13.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-02-05T06:00:14.248 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-02-05T06:00:14.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-02-05T05:57:19.785662+0000 last_deep_scrub_stamp 2024-02-05T06:00:14.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T06:00:14.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:57:19.785662+0000 2024-02-05T06:00:14.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-02-05T06:00:14.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:00:14.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:00:14.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:00:14.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:00:14.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:00:14.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:00:14.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:00:14.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:57:19.785662+0000 '>' 2024-02-05T05:57:19.785662+0000 2024-02-05T06:00:14.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:00:15.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:00:15.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:00:15.581 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:00:15.581 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:00:15.581 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:00:15.581 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:00:15.581 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:00:15.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:57:19.785662+0000 '>' 2024-02-05T05:57:19.785662+0000 2024-02-05T06:00:15.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:00:16.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:00:16.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:00:16.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:00:16.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:00:16.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:00:16.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:00:16.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:00:17.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:00:14.787616+0000 '>' 2024-02-05T05:57:19.785662+0000 2024-02-05T06:00:17.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:00:17.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-02-05T06:00:17.236 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-02-05T06:00:17.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-02-05T06:00:17.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-02-05T06:00:17.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-02-05T06:00:17.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-02-05T06:00:17.317 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-02-05T06:00:17.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=120 2024-02-05T06:00:17.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-02-05T06:00:17.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T06:00:17.344 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T06:00:17.345 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T06:00:17.345 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T06:00:17.345 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T06:00:17.345 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T06:00:17.345 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T06:00:17.345 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T06:00:17.345 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T06:00:17.345 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T06:00:17.345 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T06:00:17.345 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T06:00:17.345 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T06:00:17.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T06:00:17.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T06:00:17.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T06:00:17.386 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T06:00:17.386 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T06:00:17.386 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T06:00:17.386 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T06:00:17.386 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T06:00:17.386 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T06:00:17.387 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T06:00:17.387 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T06:00:17.387 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T06:00:17.387 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T06:00:17.387 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T06:00:17.387 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-02-05T06:00:17.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T06:00:17.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T06:00:17.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T06:00:17.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T06:00:17.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-02-05T06:00:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-02-05T06:00:17.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-02-05T06:00:17.881 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'ecpool' removed 2024-02-05T06:00:17.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:00:17.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:00:17.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:00:17.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:00:17.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:00:17.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:00:17.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:00:17.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:00:17.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:00:18.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:00:18.212 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:00:18.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:00:18.214 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:00:18.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:00:18.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:00:18.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:00:18.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:00:18.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:00:18.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:00:18.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:00:18.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:00:18.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:00:18.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:00:18.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:00:18.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:18.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:00:18.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:00:18.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:00:18.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:00:18.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:00:18.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:00:18.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:00:18.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:00:18.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:00:18.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:00:18.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:00:18.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:00:18.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:00:18.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:00:18.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:00:18.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:00:18.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:00:18.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:00:18.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:00:18.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:00:18.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:00:18.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:00:18.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:00:18.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:00:18.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:00:18.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:00:18.269 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:00:18.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:00:18.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:00:18.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:00:18.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:00:18.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:18.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:00:18.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:00:18.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:00:18.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:00:18.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:00:18.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:00:18.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:18.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:00:18.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T06:00:18.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:00:18.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:00:18.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:00:18.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:00:18.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-02-05T06:00:18.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-02-05T06:00:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-02-05T06:00:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-02-05T06:00:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-02-05T06:00:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-02-05T06:00:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-02-05T06:00:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-02-05T06:00:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:00:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:00:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:00:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:00:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:00:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T06:00:18.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:00:18.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:00:18.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:00:18.676 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:00:18.676 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:18.676 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:00:18.676 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:00:18.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T06:00:18.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:00:18.888 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:00:18.888 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:00:18.888 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:00:18.888 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:00:18.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:00:18.889 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:00:18.889 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:00:18.889 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:00:18.889 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:00:18.889 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:18.889 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:00:18.890 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:00:18.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:00:18.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T06:00:18.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:00:18.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:00:18.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:00:18.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:00:19.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:00:19.000 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:00:19.000 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:00:19.000 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:00:19.000 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:00:19.000 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:19.000 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:00:19.000 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:00:19.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:00:19.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T06:00:19.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-02-05T06:00:19.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:00:19.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:00:19.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:00:19.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:00:19.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:00:19.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:00:20.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:00:20.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:00:20.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:00:20.014 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:00:20.014 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:20.014 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:00:20.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:00:20.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:00:20.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:00:20.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-02-05T06:00:20.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-02-05T06:00:20.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-02-05T06:00:20.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:00:20.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:00:20.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:00:20.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:00:20.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:00:20.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:00:20.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:00:20.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:00:20.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:00:20.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:00:20.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:00:20.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:00:20.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:00:20.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:00:20.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:00:20.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:00:20.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:00:20.057 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:00:20.057 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:20.057 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:00:20.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:00:20.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:00:20.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:00:20.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:00:20.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:00:20.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:00:20.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:00:20.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:00:20.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:00:20.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:00:20.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:00:20.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:00:20.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:00:20.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:00:20.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:00:20.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c5f4bf3a-15bb-4966-baef-5caa09dc2d5b 2024-02-05T06:00:20.059 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 c5f4bf3a-15bb-4966-baef-5caa09dc2d5b 2024-02-05T06:00:20.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c5f4bf3a-15bb-4966-baef-5caa09dc2d5b' 2024-02-05T06:00:20.060 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:00:20.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0ecBlcYHWBRAAhEYV+sjnhFyp6Q6xdzxLJA== 2024-02-05T06:00:20.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0ecBlcYHWBRAAhEYV+sjnhFyp6Q6xdzxLJA=="}' 2024-02-05T06:00:20.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c5f4bf3a-15bb-4966-baef-5caa09dc2d5b -i td/osd-scrub-repair/0/new.json 2024-02-05T06:00:20.332 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:00:20.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:00:20.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0ecBlcYHWBRAAhEYV+sjnhFyp6Q6xdzxLJA== --osd-uuid c5f4bf3a-15bb-4966-baef-5caa09dc2d5b 2024-02-05T06:00:20.383 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:20.404+0000 7f642cc4f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:20.399 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:20.420+0000 7f642cc4f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:20.400 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:20.420+0000 7f642cc4f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:20.401 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:20.420+0000 7f642cc4f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:00:20.401 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:20.420+0000 7f642cc4f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:00:20.402 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:20.420+0000 7f642cc4f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:00:20.491 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:20.512+0000 7f642cc4f640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:00:26.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:00:26.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:00:26.861 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:00:26.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:00:26.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:00:27.248 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:00:27.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:00:27.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:00:27.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:00:27.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:00:27.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:00:27.281 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:27.300+0000 7f519212e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:27.320 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:27.336+0000 7f519212e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:27.321 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:27.336+0000 7f519212e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:27.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:00:27.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:00:27.588 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:00:27.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:00:27.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:00:27.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:00:27.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:27.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:00:27.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:27.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:27.869 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:27.888+0000 7f519212e640 -1 Falling back to public interface 2024-02-05T06:00:27.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:28.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:28.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:28.921 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:00:28.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:00:28.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:28.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:29.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:30.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:30.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:30.226 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:00:30.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:00:30.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:30.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:30.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:30.553 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:30.572+0000 7f519212e640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:00:31.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:31.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:31.544 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:00:31.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:00:31.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:31.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:31.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:32.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:32.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:32.882 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:00:32.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:00:32.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:32.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:33.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:34.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:34.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:34.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:00:34.202 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:00:34.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:34.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:34.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:35.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:35.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:35.624 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:00:35.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:00:35.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:35.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:35.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:36.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:36.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:36.928 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:00:36.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:00:36.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:36.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:37.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:38.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:38.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:38.236 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:00:38.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:00:38.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:38.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:38.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:39.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:39.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:39.617 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:00:39.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:00:39.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:39.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:39.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:40.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:40.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:40.933 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:00:40.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:00:40.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:40.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:41.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:42.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:42.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:42.380 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:00:42.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:00:42.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:42.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:42.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:43.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:43.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:43.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:00:43.702 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:00:43.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:43.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:44.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:45.022 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:00:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:00:45.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:45.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:45.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:46.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:46.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:46.820 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:00:46.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:00:46.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:46.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:47.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:48.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:48.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:48.666 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:00:48.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:00:48.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:48.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:49.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:50.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:50.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:50.545 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:00:50.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:00:50.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:50.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:51.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:52.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:52.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:52.382 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:00:52.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:00:52.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:52.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:53.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:54.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:54.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:54.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:00:54.215 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:00:54.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:54.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:54.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:55.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:55.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:55.778 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:00:55.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:00:55.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:55.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:56.095 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3366624424,v1:127.0.0.1:6803/3366624424] [v2:127.0.0.1:6804/3366624424,v1:127.0.0.1:6805/3366624424] exists,up c5f4bf3a-15bb-4966-baef-5caa09dc2d5b 2024-02-05T06:00:56.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:00:56.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:00:56.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:00:56.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-02-05T06:00:56.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-02-05T06:00:56.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:00:56.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:00:56.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:00:56.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:00:56.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:00:56.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:00:56.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:00:56.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:00:56.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:00:56.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:00:56.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:00:56.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:00:56.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:00:56.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:00:56.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:00:56.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:00:56.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:00:56.100 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:00:56.100 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:56.100 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:00:56.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:00:56.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:00:56.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:00:56.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:00:56.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:00:56.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:00:56.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:00:56.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:00:56.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:00:56.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:00:56.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:00:56.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:00:56.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:00:56.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:00:56.102 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:00:56.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=170ecd0f-2bd8-43d7-a5bc-bad18d97c676 2024-02-05T06:00:56.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 170ecd0f-2bd8-43d7-a5bc-bad18d97c676' 2024-02-05T06:00:56.102 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 170ecd0f-2bd8-43d7-a5bc-bad18d97c676 2024-02-05T06:00:56.103 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:00:56.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCYecBllrKtCBAA87GIA2msI8wlHhR26fpNlA== 2024-02-05T06:00:56.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCYecBllrKtCBAA87GIA2msI8wlHhR26fpNlA=="}' 2024-02-05T06:00:56.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 170ecd0f-2bd8-43d7-a5bc-bad18d97c676 -i td/osd-scrub-repair/1/new.json 2024-02-05T06:00:56.498 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:00:56.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T06:00:56.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYecBllrKtCBAA87GIA2msI8wlHhR26fpNlA== --osd-uuid 170ecd0f-2bd8-43d7-a5bc-bad18d97c676 2024-02-05T06:00:56.556 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:56.575+0000 7f7b747bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:56.583 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:56.603+0000 7f7b747bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:56.585 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:56.603+0000 7f7b747bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:56.585 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:56.603+0000 7f7b747bd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:00:56.586 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:56.603+0000 7f7b747bd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:00:56.586 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:56.603+0000 7f7b747bd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:00:56.659 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:00:56.679+0000 7f7b747bd640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T06:01:03.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T06:01:03.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:01:03.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:01:03.615 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:01:03.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:01:04.002 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:01:04.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:01:04.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:01:04.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:01:04.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:01:04.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:01:04.034 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:04.051+0000 7f4b526c9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:04.064 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:04.083+0000 7f4b526c9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:04.065 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:04.083+0000 7f4b526c9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:04.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:01:04.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:01:04.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:01:04.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:01:04.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:01:04.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:04.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:01:04.323 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:01:04.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:04.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:04.612 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:04.631+0000 7f4b526c9640 -1 Falling back to public interface 2024-02-05T06:01:04.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:05.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:05.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:05.649 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:01:05.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:01:05.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:05.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:05.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:06.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:06.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:06.949 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:01:06.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:01:06.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:06.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:07.282 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:07.303+0000 7f4b526c9640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:01:07.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:08.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:08.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:08.346 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:01:08.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:01:08.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:08.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:08.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:09.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:09.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:09.668 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:01:09.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:01:09.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:09.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:09.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:10.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:10.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:10.978 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:01:10.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:01:10.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:10.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:11.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:12.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:12.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:12.523 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:01:12.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:01:12.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:12.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:12.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:13.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:13.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:13.845 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:01:13.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:01:13.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:13.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:14.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:15.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:15.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:15.161 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:01:15.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:01:15.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:15.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:15.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:16.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:16.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:16.481 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:01:16.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:01:16.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:16.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:16.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:17.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:17.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:17.812 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:01:17.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:01:17.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:17.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:18.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:19.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:19.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:19.233 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:01:19.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:01:19.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:19.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:19.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:20.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:20.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:20.661 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:01:20.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:01:20.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:20.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:21.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:22.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:22.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:22.035 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:01:22.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:01:22.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:22.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:22.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:23.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:23.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:23.574 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:01:23.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:01:23.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:23.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:24.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:25.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:25.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:25.190 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:01:25.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:01:25.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:25.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:25.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:26.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:26.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:26.712 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:01:26.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:01:26.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:26.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:27.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:28.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:28.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:28.633 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:01:28.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:01:28.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:28.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:29.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:30.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:30.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:30.079 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:01:30.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:01:30.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:30.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:30.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:31.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:31.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:31.889 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:01:31.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:01:31.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:31.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:32.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:33.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:33.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:33.357 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:01:33.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:01:33.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:33.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:33.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:34.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:34.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:34.671 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T06:01:34.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T06:01:34.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:34.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:01:35.000 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3821158595,v1:127.0.0.1:6811/3821158595] [v2:127.0.0.1:6812/3821158595,v1:127.0.0.1:6813/3821158595] exists,up 170ecd0f-2bd8-43d7-a5bc-bad18d97c676 2024-02-05T06:01:35.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:01:35.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:01:35.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:01:35.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-02-05T06:01:35.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-02-05T06:01:35.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:01:35.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:01:35.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T06:01:35.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:01:35.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:01:35.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:01:35.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:01:35.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:01:35.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:01:35.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:01:35.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:01:35.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:01:35.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:01:35.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:01:35.007 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:01:35.007 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:01:35.007 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:01:35.007 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:01:35.008 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:01:35.008 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:01:35.008 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:01:35.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:01:35.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:01:35.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:01:35.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:01:35.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:01:35.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:01:35.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:01:35.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:01:35.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:01:35.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:01:35.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:01:35.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:01:35.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:01:35.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:01:35.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c4c9ead0-aeca-4c85-8b6b-39e9efae8581 2024-02-05T06:01:35.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c4c9ead0-aeca-4c85-8b6b-39e9efae8581' 2024-02-05T06:01:35.011 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 c4c9ead0-aeca-4c85-8b6b-39e9efae8581 2024-02-05T06:01:35.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:01:35.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/ecBlXgTjAhAAlwAJztu3i5zldCV1eNpt8w== 2024-02-05T06:01:35.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/ecBlXgTjAhAAlwAJztu3i5zldCV1eNpt8w=="}' 2024-02-05T06:01:35.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c4c9ead0-aeca-4c85-8b6b-39e9efae8581 -i td/osd-scrub-repair/2/new.json 2024-02-05T06:01:35.406 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:01:35.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T06:01:35.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/ecBlXgTjAhAAlwAJztu3i5zldCV1eNpt8w== --osd-uuid c4c9ead0-aeca-4c85-8b6b-39e9efae8581 2024-02-05T06:01:35.460 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:35.479+0000 7fd8558c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:35.508 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:35.527+0000 7fd8558c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:35.509 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:35.527+0000 7fd8558c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:35.510 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:35.531+0000 7fd8558c8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:01:35.510 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:35.531+0000 7fd8558c8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:01:35.510 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:35.531+0000 7fd8558c8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:01:35.583 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:35.603+0000 7fd8558c8640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T06:01:42.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T06:01:42.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:01:42.658 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T06:01:42.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T06:01:42.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:01:43.050 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T06:01:43.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T06:01:43.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:01:43.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:01:43.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:01:43.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:01:43.084 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:43.102+0000 7fd8fe042640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:43.108 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:43.126+0000 7fd8fe042640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:43.109 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:43.126+0000 7fd8fe042640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:43.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T06:01:43.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:01:43.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:01:43.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:01:43.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:01:43.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:43.375 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:01:43.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:01:43.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:43.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:01:43.668 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:43.686+0000 7fd8fe042640 -1 Falling back to public interface 2024-02-05T06:01:43.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:44.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:44.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:44.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:01:44.691 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:01:44.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:44.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:01:44.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:46.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:46.000 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:01:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:01:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:46.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:01:46.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:46.329 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:01:46.346+0000 7fd8fe042640 -1 osd.2 0 log_to_monitors true 2024-02-05T06:01:47.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:47.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:47.308 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:01:47.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:01:47.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:47.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:01:47.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:48.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:48.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:48.687 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:01:48.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:01:48.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:48.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:01:49.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:50.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:50.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:50.007 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:01:50.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:01:50.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:50.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:01:50.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:51.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:51.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:51.326 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:01:51.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:01:51.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:51.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:01:51.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:52.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:52.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:52.823 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:01:52.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:01:52.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:52.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:01:53.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:54.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:54.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:54.149 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:01:54.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:01:54.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:54.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:01:54.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:55.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:55.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:55.520 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:01:55.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:01:55.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:55.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:01:55.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:56.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:56.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:56.906 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:01:56.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:01:56.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:56.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:01:57.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:58.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:58.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:58.317 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:01:58.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:01:58.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:58.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:01:58.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:59.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:59.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:59.636 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:01:59.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:01:59.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:59.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:01:59.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:00.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:00.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:00.954 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:02:00.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:02:00.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:00.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:02:01.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:02.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:02.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:02.519 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:02:02.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:02:02.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:02.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:02:02.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:03.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:03.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:03.920 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:02:03.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:02:03.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:03.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:02:04.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:05.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:05.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:05.553 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:02:05.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:02:05.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:05.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:02:06.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:07.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:07.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:07.665 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:02:07.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:02:07.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:07.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:02:08.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:09.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:09.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:09.387 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:02:09.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:02:09.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:09.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:02:09.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:10.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:10.703 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:02:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:02:10.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:10.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:02:11.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:12.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:12.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:12.185 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:02:12.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:02:12.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:12.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:02:12.509 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2318926934,v1:127.0.0.1:6819/2318926934] [v2:127.0.0.1:6820/2318926934,v1:127.0.0.1:6821/2318926934] exists,up c4c9ead0-aeca-4c85-8b6b-39e9efae8581 2024-02-05T06:02:12.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:02:12.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:02:12.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:02:12.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-02-05T06:02:12.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T06:02:12.822 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T06:02:12.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T06:02:12.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T06:02:13.345 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T06:02:13.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:02:14.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T06:02:16.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-02-05T06:02:16.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-02-05T06:02:17.141 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'foo' created 2024-02-05T06:02:17.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:02:18.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-02-05T06:02:18.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T06:02:18.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T06:02:18.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-02-05T06:02:18.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T06:02:18.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-02-05T06:02:18.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T06:02:18.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T06:02:19.593 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'ecpool' created 2024-02-05T06:02:19.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:02:20.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-02-05T06:02:20.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-02-05T06:02:21.198 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 3 allow_ec_overwrites to true 2024-02-05T06:02:21.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T06:02:21.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:02:21.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:02:21.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:02:21.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:02:21.224 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:02:21.224 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:02:21.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:02:21.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:02:21.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:02:21.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:02:21.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:02:21.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:02:21.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:02:21.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:02:21.347 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:02:21.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:02:21.754 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:02:21.754 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:02:21.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:02:21.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:02:21.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:02:22.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-02-05T06:02:22.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-02-05T06:02:22.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-02-05T06:02:22.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:02:22.093 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:02:22.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-02-05T06:02:22.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-02-05T06:02:22.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672977' 2024-02-05T06:02:22.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:02:22.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:02:22.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-02-05T06:02:22.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-02-05T06:02:22.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672977 2-64424509445' 2024-02-05T06:02:22.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:02:22.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-02-05T06:02:22.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:02:22.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:02:22.424 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-02-05T06:02:22.424 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:02:22.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-02-05T06:02:22.426 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836503 2024-02-05T06:02:22.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-02-05T06:02:22.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:02:22.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-02-05T06:02:22.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:02:22.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-02-05T06:02:22.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:02:22.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:02:22.744 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-02-05T06:02:22.744 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:02:22.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-02-05T06:02:22.746 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672977 2024-02-05T06:02:22.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-02-05T06:02:22.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:02:23.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-02-05T06:02:23.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:02:23.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-02-05T06:02:23.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:02:23.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:02:23.086 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-02-05T06:02:23.086 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:02:23.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-02-05T06:02:23.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-02-05T06:02:23.088 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509445 2024-02-05T06:02:23.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:02:23.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-02-05T06:02:23.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:02:23.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:02:23.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:02:23.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:02:23.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:02:23.809 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:02:23.809 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:02:23.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:02:23.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:02:23.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:02:23.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:02:24.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:02:24.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:02:24.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:02:24.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:02:24.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:02:24.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:02:24.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:02:24.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T06:02:24.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-02-05T06:02:24.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:02:24.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:02:24.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:02:24.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:02:24.506 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:02:24.506 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:02:24.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:02:24.507 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:02:24.507 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:02:24.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:02:24.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:02:24.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:02:24.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:02:24.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:02:24.604 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:02:24.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:02:24.895 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:02:24.895 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:02:24.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:02:24.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:02:24.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:02:25.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-02-05T06:02:25.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-02-05T06:02:25.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-02-05T06:02:25.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:02:25.079 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:02:25.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-02-05T06:02:25.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-02-05T06:02:25.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672979' 2024-02-05T06:02:25.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:02:25.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:02:25.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-02-05T06:02:25.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-02-05T06:02:25.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672979 2-64424509447' 2024-02-05T06:02:25.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:02:25.383 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-02-05T06:02:25.383 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:02:25.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:02:25.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-02-05T06:02:25.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:02:25.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-02-05T06:02:25.386 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836504 2024-02-05T06:02:25.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-02-05T06:02:25.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:02:25.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-02-05T06:02:25.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:02:26.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:02:26.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:02:27.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836504 2024-02-05T06:02:27.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:02:27.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:02:27.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-02-05T06:02:27.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:02:27.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-02-05T06:02:27.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:02:27.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-02-05T06:02:27.071 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672979 2024-02-05T06:02:27.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-02-05T06:02:27.072 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:02:27.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-02-05T06:02:27.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:02:27.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:02:27.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-02-05T06:02:27.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:02:27.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-02-05T06:02:27.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:02:27.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-02-05T06:02:27.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-02-05T06:02:27.418 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509447 2024-02-05T06:02:27.418 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:02:27.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-02-05T06:02:27.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:02:27.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:02:27.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:02:28.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:02:28.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:02:28.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:02:28.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:02:28.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:02:28.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:02:28.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:02:28.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:02:28.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:02:28.436 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:02:28.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:02:28.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:02:28.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:02:28.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:02:28.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:02:28.821 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-02-05T06:02:28.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T06:02:28.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-02-05T06:02:28.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-02-05T06:02:28.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:02:28.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:02:28.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-02-05T06:02:28.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:02:28.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:02:28.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:02:29.261 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:02:29.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:02:29.848 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:02:29.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:02:29.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:02:29.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:02:30.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-02-05T06:02:30.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T06:02:30.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T06:02:30.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-02-05T06:02:30.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-02-05T06:02:30.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:02:30.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:02:30.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:02:30.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:02:30.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:02:30.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:02:30.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:02:30.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:02:30.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:02:30.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:02:30.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:02:30.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:02:30.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:02:30.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:02:30.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:02:30.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:02:30.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:02:30.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:02:30.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:02:30.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:02:30.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:02:30.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:02:30.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:02:30.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:02:32.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:02:32.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:02:32.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:02:32.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:02:32.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:02:32.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:02:32.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:02:32.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:02:32.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:02:32.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:02:32.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:02:32.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:02:32.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:02:32.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:02:32.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:02:32.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:02:32.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:02:32.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:02:32.913 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:02:32.913 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:02:32.913 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:02:32.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:02:32.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:02:32.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:02:32.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:02:32.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:02:32.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:02:32.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:02:32.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:02:32.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:02:32.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:02:32.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:02:32.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:02:32.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:02:32.914 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:02:32.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:02:32.915 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:02:32.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:02:32.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:02:32.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:02:32.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:02:32.945 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:02:32.969+0000 7f15a6dda640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:32.948 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:02:32.969+0000 7f15a6dda640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:32.949 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:02:32.973+0000 7f15a6dda640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:33.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:02:33.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:02:33.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:02:33.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:02:33.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:02:33.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:33.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:02:33.244 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:02:33.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:33.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:33.476 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:02:33.497+0000 7f15a6dda640 -1 Falling back to public interface 2024-02-05T06:02:33.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:34.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:34.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:34.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:02:34.581 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:02:34.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:34.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:34.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:35.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:35.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:35.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:02:35.900 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:02:35.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:35.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:36.201 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:02:36.225+0000 7f15a6dda640 -1 osd.1 35 log_to_monitors true 2024-02-05T06:02:36.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:37.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:37.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:37.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:02:37.214 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:02:37.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:37.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:37.741 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/1289810907,v1:127.0.0.1:6811/1289810907] [v2:127.0.0.1:6812/1289810907,v1:127.0.0.1:6813/1289810907] exists,up 170ecd0f-2bd8-43d7-a5bc-bad18d97c676 2024-02-05T06:02:37.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:02:37.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:02:37.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:02:37.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:02:37.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:02:37.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:02:37.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:02:37.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:02:37.743 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:02:37.743 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:02:37.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:02:37.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:02:37.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:02:37.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:02:37.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:02:37.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:02:37.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:02:37.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:02:37.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:02:38.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:02:38.188 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:02:38.188 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:02:38.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:02:38.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:02:38.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:02:38.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-02-05T06:02:38.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-02-05T06:02:38.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-02-05T06:02:38.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:02:38.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:02:38.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-02-05T06:02:38.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-02-05T06:02:38.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-167503724546' 2024-02-05T06:02:38.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:02:38.480 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:02:38.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-02-05T06:02:38.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-02-05T06:02:38.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-167503724546 2-64424509451' 2024-02-05T06:02:38.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:02:38.642 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-02-05T06:02:38.642 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:02:38.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:02:38.644 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-02-05T06:02:38.644 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:02:38.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-02-05T06:02:38.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-02-05T06:02:38.645 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836508 2024-02-05T06:02:38.646 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:02:38.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-02-05T06:02:38.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:02:38.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:02:38.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724546 2024-02-05T06:02:38.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:02:38.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724546 2024-02-05T06:02:38.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:02:38.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-02-05T06:02:38.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2024-02-05T06:02:38.987 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 167503724546 2024-02-05T06:02:38.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:02:39.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-02-05T06:02:39.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:02:39.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-02-05T06:02:39.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:02:39.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:02:39.338 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-02-05T06:02:39.338 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:02:39.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-02-05T06:02:39.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-02-05T06:02:39.339 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509451 2024-02-05T06:02:39.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:02:39.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509451 2024-02-05T06:02:39.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:02:40.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:02:40.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:02:40.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-02-05T06:02:40.966 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:02:40.966 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:02:40.967 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:02:41.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:02:41.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:02:41.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:02:41.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:02:41.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:02:41.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:02:41.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:02:41.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:02:41.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T06:02:41.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:02:41.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:02:41.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:02:42.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-02-05T06:02:42.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-02-05T06:02:42.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:02:42.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-02-05T06:02:42.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:02:42.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:02:42.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:02:42.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:02:42.146 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:02:42.146 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:02:42.146 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:02:42.146 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:02:42.146 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:02:42.147 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:02:42.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T06:02:42.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:02:42.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:02:42.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:02:42.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-02-05T06:02:42.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-02-05T06:02:42.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:02:42.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:02:42.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:02:42.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:02:42.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:02:42.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:02:42.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:02:43.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:02:43.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:02:43.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:02:43.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:02:43.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:02:43.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:02:43.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:02:43.405 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:02:43.405 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:02:43.405 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:02:43.405 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:02:43.406 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:02:43.406 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:02:43.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:02:43.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:02:43.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:02:43.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:02:44.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:02:44.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:02:44.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:02:44.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T06:02:44.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-02-05T06:02:44.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-02-05T06:02:44.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:02:44.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:02:44.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-02-05T06:02:44.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:02:44.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:02:44.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:02:44.507 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:02:44.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:02:45.102 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:02:45.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:02:45.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:02:45.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:02:45.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-02-05T06:02:45.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-02-05T06:02:45.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T06:02:45.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-02-05T06:02:45.464 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records in 2024-02-05T06:02:45.464 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records out 2024-02-05T06:02:45.464 INFO:tasks.workunit.client.0.smithi148.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000177132 s, 11.6 MB/s 2024-02-05T06:02:45.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:02:45.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:02:45.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:02:45.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:02:45.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:02:45.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:02:45.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:02:45.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:02:45.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:02:45.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:02:45.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:02:45.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:02:45.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:02:45.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:02:45.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:02:45.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:02:46.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:02:46.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:02:46.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:02:46.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:02:46.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:02:46.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:02:46.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:02:46.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:02:48.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:02:48.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:02:48.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:02:48.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:02:48.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:02:48.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:02:48.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:02:48.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:02:48.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:02:48.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:02:48.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:02:48.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:02:48.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:02:48.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:02:48.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:02:48.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:02:48.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:02:48.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:02:48.831 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:02:48.831 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:02:48.831 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:02:48.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:02:48.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:02:48.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:02:48.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:02:48.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:02:48.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:02:48.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:02:48.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:02:48.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:02:48.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:02:48.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:02:48.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:02:48.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:02:48.833 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:02:48.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:02:48.833 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:02:48.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:02:48.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:02:48.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:02:48.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:02:48.866 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:02:48.889+0000 7ff98a11d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:48.870 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:02:48.893+0000 7ff98a11d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:48.871 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:02:48.893+0000 7ff98a11d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:49.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:02:49.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:02:49.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:02:49.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:02:49.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:02:49.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:49.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:02:49.181 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:02:49.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:49.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:02:49.411 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:02:49.433+0000 7ff98a11d640 -1 Falling back to public interface 2024-02-05T06:02:49.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:50.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:50.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:50.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:02:50.514 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:02:50.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:50.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:02:50.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:51.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:51.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:51.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:02:51.828 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:02:51.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:51.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:02:52.018 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:02:52.041+0000 7ff98a11d640 -1 osd.0 43 log_to_monitors true 2024-02-05T06:02:52.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:53.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:53.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:53.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:02:53.151 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:02:53.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:53.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:02:53.564 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 48 up_thru 48 down_at 44 last_clean_interval [5,43) [v2:127.0.0.1:6802/1979626016,v1:127.0.0.1:6803/1979626016] [v2:127.0.0.1:6804/1979626016,v1:127.0.0.1:6805/1979626016] exists,up c5f4bf3a-15bb-4966-baef-5caa09dc2d5b 2024-02-05T06:02:53.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:02:53.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:02:53.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:02:53.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:02:53.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:02:53.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:02:53.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:02:53.565 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:02:53.565 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:02:53.565 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:02:53.566 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:02:53.566 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:02:53.566 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:02:53.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:02:53.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:02:53.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:02:53.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:02:53.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:02:53.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:02:53.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:02:53.996 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:02:53.996 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:02:53.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:02:53.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:02:53.996 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:02:54.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430210 2024-02-05T06:02:54.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430210 2024-02-05T06:02:54.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210' 2024-02-05T06:02:54.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:02:54.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:02:54.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724550 2024-02-05T06:02:54.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724550 2024-02-05T06:02:54.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-167503724550' 2024-02-05T06:02:54.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:02:54.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:02:54.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-02-05T06:02:54.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-02-05T06:02:54.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-167503724550 2-64424509455' 2024-02-05T06:02:54.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:02:54.623 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430210 2024-02-05T06:02:54.623 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:02:54.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:02:54.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430210 2024-02-05T06:02:54.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:02:54.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430210 2024-02-05T06:02:54.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430210' 2024-02-05T06:02:54.626 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 206158430210 2024-02-05T06:02:54.626 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:02:54.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430210 -lt 206158430210 2024-02-05T06:02:54.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:02:54.941 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724550 2024-02-05T06:02:54.942 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:02:54.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:02:54.944 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724550 2024-02-05T06:02:54.944 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:02:54.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724550 2024-02-05T06:02:54.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724550' 2024-02-05T06:02:54.946 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 167503724550 2024-02-05T06:02:54.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:02:55.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724550 -lt 167503724550 2024-02-05T06:02:55.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:02:55.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-02-05T06:02:55.303 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:02:55.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:02:55.306 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-02-05T06:02:55.306 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:02:55.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-02-05T06:02:55.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-02-05T06:02:55.307 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509455 2024-02-05T06:02:55.308 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:02:55.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509455 2024-02-05T06:02:55.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:02:56.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:02:56.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:02:56.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-02-05T06:02:56.915 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:02:56.915 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:02:56.915 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:02:57.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:02:57.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:02:57.318 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:02:57.318 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:02:57.318 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:02:57.318 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:02:57.318 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:02:57.318 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:02:57.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:02:57.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:02:57.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:02:57.662 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:02:58.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:02:58.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:02:58.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:02:58.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T06:02:58.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-02-05T06:02:58.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-02-05T06:02:58.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:02:58.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:02:58.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-02-05T06:02:58.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:02:58.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:02:58.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:02:58.581 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:02:58.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:02:59.058 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:02:59.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:02:59.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:02:59.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:02:59.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-02-05T06:02:59.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T06:02:59.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T06:02:59.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-02-05T06:02:59.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:02:59.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:02:59.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:02:59.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:02:59.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-02-05T06:02:59.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:02:59.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:02:59.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:02:59.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:02:59.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:02:59.343 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:02:59.343 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:02:59.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:02:59.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:02:59.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:03:00.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:03:00.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-02-05T06:03:00.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:03:00.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:03:00.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:03:00.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:03:00.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:03:00.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-02-05T06:03:01.914 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-02-05T06:03:02.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:03:02.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:03:02.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:03:02.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:03:02.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:03:02.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:03:02.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:03:02.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:03:02.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:03:02.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:03:02.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:03:02.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:03:02.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:03:02.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:03:02.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:03:02.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:03:02.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:03:02.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:03:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:03:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:03:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:03:02.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:03:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:03:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:03:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:03:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:03:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:03:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:03:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:03:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:03:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:03:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:03:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:03:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:03:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:03:02.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:03:02.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:03:02.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:03:02.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:03:02.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:03:02.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:03:02.490 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:02.512+0000 7f5a4b6d0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:02.491 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:02.512+0000 7f5a4b6d0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:02.493 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:02.516+0000 7f5a4b6d0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:02.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:03:02.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:03:02.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:03:02.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:03:02.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:03:02.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:02.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:03:02.789 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:03:02.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:02.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:03.028 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:03.048+0000 7f5a4b6d0640 -1 Falling back to public interface 2024-02-05T06:03:03.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:04.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:04.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:04.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:03:04.121 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:03:04.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:04.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:04.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:05.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:05.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:05.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:03:05.438 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:03:05.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:05.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:05.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:05.804 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:05.824+0000 7f5a4b6d0640 -1 osd.1 52 log_to_monitors true 2024-02-05T06:03:06.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:06.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:06.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:03:06.749 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:03:06.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:06.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:07.173 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 53 last_clean_interval [39,52) [v2:127.0.0.1:6810/160775470,v1:127.0.0.1:6811/160775470] [v2:127.0.0.1:6812/160775470,v1:127.0.0.1:6813/160775470] exists,up 170ecd0f-2bd8-43d7-a5bc-bad18d97c676 2024-02-05T06:03:07.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:03:07.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:03:07.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:03:07.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:03:07.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:03:07.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:03:07.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:03:07.174 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:03:07.174 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:03:07.174 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:03:07.174 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:03:07.175 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:03:07.175 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:03:07.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:03:07.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:03:07.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:03:07.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:03:07.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:03:07.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:03:07.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:03:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:03:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:03:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:03:07.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:03:07.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:03:07.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430214 2024-02-05T06:03:07.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430214 2024-02-05T06:03:07.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214' 2024-02-05T06:03:07.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:03:07.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:03:08.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168578 2024-02-05T06:03:08.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168578 2024-02-05T06:03:08.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214 1-240518168578' 2024-02-05T06:03:08.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:03:08.194 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:03:08.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-02-05T06:03:08.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-02-05T06:03:08.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214 1-240518168578 2-64424509459' 2024-02-05T06:03:08.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:03:08.352 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430214 2024-02-05T06:03:08.352 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:03:08.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:03:08.354 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430214 2024-02-05T06:03:08.354 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:03:08.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430214 2024-02-05T06:03:08.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430214' 2024-02-05T06:03:08.355 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 206158430214 2024-02-05T06:03:08.355 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:03:08.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430212 -lt 206158430214 2024-02-05T06:03:08.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:03:09.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:03:09.687 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:03:10.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430214 -lt 206158430214 2024-02-05T06:03:10.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:03:10.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:03:10.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-240518168578 2024-02-05T06:03:10.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:03:10.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-240518168578 2024-02-05T06:03:10.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:03:10.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168578 2024-02-05T06:03:10.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 240518168578' 2024-02-05T06:03:10.008 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 240518168578 2024-02-05T06:03:10.008 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:03:10.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168578 -lt 240518168578 2024-02-05T06:03:10.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:03:10.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-02-05T06:03:10.342 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:03:10.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:03:10.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-02-05T06:03:10.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:03:10.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-02-05T06:03:10.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-02-05T06:03:10.346 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509459 2024-02-05T06:03:10.346 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:03:10.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-02-05T06:03:10.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:03:10.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:03:10.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:03:11.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:03:11.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:03:11.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:03:11.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:03:11.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:03:11.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:03:11.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:03:11.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:03:11.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:03:11.367 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:03:11.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:03:11.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:03:11.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T06:03:11.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-02-05T06:03:11.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:03:11.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T06:03:11.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:03:11.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:03:11.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:03:11.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:03:11.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:03:11.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:03:11.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:03:11.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:03:11.828 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:03:11.828 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:03:12.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:03:12.116 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:03:12.117 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:03:12.117 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:03:12.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T06:03:12.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T06:03:12.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:03:12.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:03:12.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:03:12.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:03:12.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:03:12.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:03:12.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:03:12.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:03:12.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:03:12.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:03:12.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:03:12.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:03:13.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:03:13.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:03:13.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:03:13.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:03:13.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:03:13.085 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:03:13.085 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:03:13.085 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:03:13.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:03:13.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:03:13.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:03:13.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:03:13.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T06:03:13.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 3 2024-02-05T06:03:13.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:03:13.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T06:03:13.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:03:13.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:03:13.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:03:13.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:03:13.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:03:13.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:03:13.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:03:13.854 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:03:13.854 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:03:13.854 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:03:14.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:03:14.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:03:14.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:03:14.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:03:14.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T06:03:14.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T06:03:14.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:03:14.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:03:14.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:03:14.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:03:14.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:03:14.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:03:14.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:03:14.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:03:14.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:03:14.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:03:14.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:03:14.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:03:15.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:03:15.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:03:15.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:03:15.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:03:15.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:03:15.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:03:15.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:03:15.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:03:15.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:03:15.451 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:03:15.451 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:03:15.451 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:03:15.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:03:15.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:03:15.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:03:15.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T06:03:15.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-02-05T06:03:15.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-02-05T06:03:15.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:03:15.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:03:15.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-02-05T06:03:15.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:03:15.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:03:15.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:03:16.306 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:03:16.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:03:16.858 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:03:16.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:03:16.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:03:16.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:03:17.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-02-05T06:03:17.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-02-05T06:03:17.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T06:03:17.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-02-05T06:03:17.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-02-05T06:03:17.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-02-05T06:03:17.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T06:03:17.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:03:17.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:03:17.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:03:17.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:03:17.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T06:03:17.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:03:17.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:03:17.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:03:17.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:03:17.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:03:17.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:03:17.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:03:17.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:03:17.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:03:17.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:03:17.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:03:18.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T06:03:18.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:03:18.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:03:18.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:03:18.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:03:18.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:03:18.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T06:03:20.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:03:20.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:03:20.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:03:20.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:03:20.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:03:20.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:03:20.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:03:20.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:03:20.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:03:20.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:03:20.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:03:20.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:03:20.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:03:20.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:03:20.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:03:20.519 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:03:20.519 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:03:20.519 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:03:20.519 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:03:20.519 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:03:20.520 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:03:20.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:03:20.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:03:20.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:03:20.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:03:20.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:03:20.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:03:20.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:03:20.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:03:20.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:03:20.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:03:20.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:03:20.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:03:20.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:03:20.523 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:03:20.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:03:20.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:03:20.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:03:20.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:03:20.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:03:20.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:03:20.554 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:20.576+0000 7f09a50df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:20.557 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:20.580+0000 7f09a50df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:20.559 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:20.580+0000 7f09a50df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:20.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:03:20.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:03:20.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:03:20.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:03:20.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:03:20.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:20.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:03:20.841 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:03:20.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:20.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:21.095 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:21.116+0000 7f09a50df640 -1 Falling back to public interface 2024-02-05T06:03:21.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:22.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:22.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:22.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:03:22.200 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:03:22.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:22.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:22.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:23.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:23.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:23.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:03:23.509 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:03:23.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:23.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:23.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:24.001 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:24.024+0000 7f09a50df640 -1 osd.0 61 log_to_monitors true 2024-02-05T06:03:24.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:24.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:24.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:03:24.826 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:03:24.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:24.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:25.428 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 66 up_thru 66 down_at 62 last_clean_interval [48,61) [v2:127.0.0.1:6802/3776286746,v1:127.0.0.1:6803/3776286746] [v2:127.0.0.1:6804/3776286746,v1:127.0.0.1:6805/3776286746] exists,up c5f4bf3a-15bb-4966-baef-5caa09dc2d5b 2024-02-05T06:03:25.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:03:25.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:03:25.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:03:25.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:03:25.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:03:25.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:03:25.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:03:25.429 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:03:25.429 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:03:25.429 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:03:25.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:03:25.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:03:25.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:03:25.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:03:25.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:03:25.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:03:25.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:03:25.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:03:25.553 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:03:26.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:03:26.146 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:03:26.147 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:03:26.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:03:26.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:03:26.147 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:03:26.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-02-05T06:03:26.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-02-05T06:03:26.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538' 2024-02-05T06:03:26.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:03:26.344 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:03:26.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168583 2024-02-05T06:03:26.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168583 2024-02-05T06:03:26.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-240518168583' 2024-02-05T06:03:26.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:03:26.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:03:26.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-02-05T06:03:26.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-02-05T06:03:26.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-240518168583 2-64424509463' 2024-02-05T06:03:26.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:03:26.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841538 2024-02-05T06:03:26.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:03:26.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:03:26.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841538 2024-02-05T06:03:26.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:03:26.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-02-05T06:03:26.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841538' 2024-02-05T06:03:26.642 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 283467841538 2024-02-05T06:03:26.642 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:03:26.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841538 -lt 283467841538 2024-02-05T06:03:26.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:03:26.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-240518168583 2024-02-05T06:03:26.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:03:26.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:03:26.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-240518168583 2024-02-05T06:03:26.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:03:26.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168583 2024-02-05T06:03:26.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 240518168583' 2024-02-05T06:03:26.978 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 240518168583 2024-02-05T06:03:26.979 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:03:27.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168583 -lt 240518168583 2024-02-05T06:03:27.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:03:27.286 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-02-05T06:03:27.286 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:03:27.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:03:27.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-02-05T06:03:27.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:03:27.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-02-05T06:03:27.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-02-05T06:03:27.291 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 64424509463 2024-02-05T06:03:27.291 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:03:27.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509463 2024-02-05T06:03:27.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:03:28.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:03:28.616 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:03:28.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509463 2024-02-05T06:03:28.922 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:03:28.922 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:03:28.923 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:03:29.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:03:29.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:03:29.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:03:29.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:03:29.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:03:29.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:03:29.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:03:29.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:03:29.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:03:29.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:03:29.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:03:29.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:03:30.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:03:30.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:03:30.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:03:30.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T06:03:30.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:03:30.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:03:30.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T06:03:30.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:03:30.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T06:03:30.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:03:30.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:03:30.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T06:03:30.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:03:30.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T06:03:30.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:03:30.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:03:30.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:03:30.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:03:30.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:03:30.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:03:30.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T06:03:30.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:03:30.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:03:30.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T06:03:30.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:03:30.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:03:30.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T06:03:32.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T06:03:32.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:03:32.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:03:32.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T06:03:32.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:03:32.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:03:32.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:03:32.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:03:32.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:03:32.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:03:32.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:03:32.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:03:32.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:03:32.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:03:32.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:03:32.758 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:03:32.758 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:03:32.758 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:03:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:03:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:03:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:03:32.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:03:32.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:03:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:03:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:03:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:03:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:03:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:03:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:03:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:03:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:03:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:03:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:03:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T06:03:32.761 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.2 2024-02-05T06:03:32.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:03:32.762 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T06:03:32.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T06:03:32.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:03:32.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:03:32.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:03:32.795 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:32.816+0000 7fce5f63d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:32.797 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:32.820+0000 7fce5f63d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:32.799 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:32.820+0000 7fce5f63d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:33.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T06:03:33.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:03:33.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:03:33.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:03:33.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:03:33.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:33.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:03:33.080 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:03:33.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:33.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:03:33.339 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:33.360+0000 7fce5f63d640 -1 Falling back to public interface 2024-02-05T06:03:33.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:34.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:34.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:34.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:03:34.414 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:03:34.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:34.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:03:34.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:35.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:35.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:35.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:03:35.736 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:03:35.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:35.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:03:35.978 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:36.000+0000 7fce5f63d640 -1 osd.2 68 log_to_monitors true 2024-02-05T06:03:36.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:37.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:37.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:37.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:03:37.057 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:03:37.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:37.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:03:37.378 INFO:tasks.workunit.client.0.smithi148.stderr:osd.2 up in weight 1 up_from 73 up_thru 73 down_at 69 last_clean_interval [15,68) [v2:127.0.0.1:6818/2974091852,v1:127.0.0.1:6819/2974091852] [v2:127.0.0.1:6820/2974091852,v1:127.0.0.1:6821/2974091852] exists,up c4c9ead0-aeca-4c85-8b6b-39e9efae8581 2024-02-05T06:03:37.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:03:37.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:03:37.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:03:37.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:03:37.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:03:37.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:03:37.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:03:37.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:03:37.380 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:03:37.380 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:03:37.380 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:03:37.380 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:03:37.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:03:37.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:03:37.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:03:37.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:03:37.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:03:37.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:03:37.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:03:37.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:03:37.883 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:03:37.883 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:03:37.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:03:37.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:03:37.884 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:03:38.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841541 2024-02-05T06:03:38.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841541 2024-02-05T06:03:38.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541' 2024-02-05T06:03:38.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:03:38.040 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:03:38.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168586 2024-02-05T06:03:38.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168586 2024-02-05T06:03:38.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541 1-240518168586' 2024-02-05T06:03:38.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:03:38.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:03:38.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-02-05T06:03:38.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-02-05T06:03:38.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541 1-240518168586 2-313532612610' 2024-02-05T06:03:38.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:03:38.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841541 2024-02-05T06:03:38.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:03:38.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:03:38.329 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841541 2024-02-05T06:03:38.329 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:03:38.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841541 2024-02-05T06:03:38.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841541' 2024-02-05T06:03:38.331 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 283467841541 2024-02-05T06:03:38.331 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:03:38.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841540 -lt 283467841541 2024-02-05T06:03:38.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:03:39.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:03:39.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:03:39.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841541 -lt 283467841541 2024-02-05T06:03:39.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:03:39.948 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-240518168586 2024-02-05T06:03:39.948 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:03:39.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:03:39.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-240518168586 2024-02-05T06:03:39.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:03:39.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168586 2024-02-05T06:03:39.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 240518168586' 2024-02-05T06:03:39.952 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 240518168586 2024-02-05T06:03:39.952 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:03:40.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168586 -lt 240518168586 2024-02-05T06:03:40.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:03:40.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:03:40.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612610 2024-02-05T06:03:40.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:03:40.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612610 2024-02-05T06:03:40.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:03:40.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-02-05T06:03:40.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-02-05T06:03:40.280 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 313532612610 2024-02-05T06:03:40.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:03:40.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612610 -lt 313532612610 2024-02-05T06:03:40.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:03:40.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:03:40.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:03:41.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:03:41.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:03:41.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:03:41.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:03:41.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:03:41.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:03:41.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:03:41.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:03:41.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:03:41.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:03:41.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:03:41.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:03:41.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:03:41.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:03:41.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:03:41.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-02-05T06:03:41.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T06:03:41.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:03:41.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:03:41.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T06:03:41.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:03:41.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T06:03:41.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:03:41.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:03:41.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T06:03:41.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:03:41.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T06:03:41.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:03:41.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:03:41.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:03:41.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:03:41.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:03:41.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:03:41.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T06:03:41.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:03:41.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:03:42.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T06:03:42.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:03:42.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:03:42.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T06:03:44.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T06:03:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:03:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:03:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T06:03:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:03:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:03:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:03:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:03:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:03:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:03:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:03:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:03:44.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:03:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:03:44.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:03:44.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:03:44.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:03:44.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:03:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:03:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:03:44.301 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:03:44.301 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:03:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:03:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:03:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:03:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:03:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:03:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:03:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:03:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:03:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:03:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:03:44.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:03:44.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T06:03:44.303 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.2 2024-02-05T06:03:44.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:03:44.303 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T06:03:44.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T06:03:44.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:03:44.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:03:44.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:03:44.337 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:44.359+0000 7fa4dccb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:44.340 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:44.359+0000 7fa4dccb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:44.341 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:44.363+0000 7fa4dccb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:44.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T06:03:44.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:03:44.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:03:44.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:03:44.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:03:44.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:44.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:03:44.623 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:03:44.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:44.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:03:44.887 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:44.907+0000 7fa4dccb5640 -1 Falling back to public interface 2024-02-05T06:03:44.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:45.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:45.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:45.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:03:45.939 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:03:45.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:45.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:03:46.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:47.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:47.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:47.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:03:47.244 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:03:47.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:47.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:03:47.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:47.651 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:47.671+0000 7fa4dccb5640 -1 osd.2 75 log_to_monitors true 2024-02-05T06:03:48.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:48.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:48.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:03:48.629 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:03:48.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:48.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:03:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:osd.2 up in weight 1 up_from 80 up_thru 80 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/1712465309,v1:127.0.0.1:6819/1712465309] [v2:127.0.0.1:6820/1712465309,v1:127.0.0.1:6821/1712465309] exists,up c4c9ead0-aeca-4c85-8b6b-39e9efae8581 2024-02-05T06:03:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:03:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:03:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:03:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:03:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:03:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:03:49.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:03:49.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:03:49.108 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:03:49.108 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:03:49.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:03:49.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:03:49.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:03:49.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:03:49.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:03:49.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:03:49.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:03:49.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:03:49.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:03:49.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:03:49.660 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:03:49.661 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:03:49.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:03:49.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:03:49.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:03:49.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841545 2024-02-05T06:03:49.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841545 2024-02-05T06:03:49.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545' 2024-02-05T06:03:49.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:03:49.811 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:03:49.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168589 2024-02-05T06:03:49.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168589 2024-02-05T06:03:49.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545 1-240518168589' 2024-02-05T06:03:49.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:03:49.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:03:50.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-02-05T06:03:50.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-02-05T06:03:50.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545 1-240518168589 2-343597383682' 2024-02-05T06:03:50.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:03:50.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841545 2024-02-05T06:03:50.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:03:50.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:03:50.090 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841545 2024-02-05T06:03:50.090 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:03:50.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841545 2024-02-05T06:03:50.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841545' 2024-02-05T06:03:50.091 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 283467841545 2024-02-05T06:03:50.092 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:03:50.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841543 -lt 283467841545 2024-02-05T06:03:50.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:03:51.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:03:51.386 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:03:51.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841545 -lt 283467841545 2024-02-05T06:03:51.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:03:51.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-240518168589 2024-02-05T06:03:51.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:03:51.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:03:51.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-240518168589 2024-02-05T06:03:51.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:03:51.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168589 2024-02-05T06:03:51.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 240518168589' 2024-02-05T06:03:51.703 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 240518168589 2024-02-05T06:03:51.704 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:03:52.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168589 -lt 240518168589 2024-02-05T06:03:52.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:03:52.039 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383682 2024-02-05T06:03:52.040 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:03:52.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:03:52.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383682 2024-02-05T06:03:52.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:03:52.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-02-05T06:03:52.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383682' 2024-02-05T06:03:52.043 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 343597383682 2024-02-05T06:03:52.044 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:03:52.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383682 -lt 343597383682 2024-02-05T06:03:52.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:03:52.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:03:52.380 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:03:52.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:03:52.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:03:52.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:03:52.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:03:52.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:03:52.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:03:52.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:03:52.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:03:53.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:03:53.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:03:53.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:03:53.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:03:53.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:03:53.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:03:53.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:03:53.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-02-05T06:03:53.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T06:03:53.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-02-05T06:03:53.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-02-05T06:03:53.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:03:53.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:03:53.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-02-05T06:03:53.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:03:53.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:03:53.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:03:53.903 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:03:53.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:03:54.632 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:03:54.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:03:54.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:03:54.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:03:54.927 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-02-05T06:03:54.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T06:03:54.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T06:03:54.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-02-05T06:03:54.931 INFO:tasks.workunit.client.0.smithi148.stderr:2+0 records in 2024-02-05T06:03:54.931 INFO:tasks.workunit.client.0.smithi148.stderr:2+0 records out 2024-02-05T06:03:54.931 INFO:tasks.workunit.client.0.smithi148.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000225225 s, 18.2 MB/s 2024-02-05T06:03:54.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:03:54.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:03:54.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:03:54.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:03:54.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:03:54.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:03:54.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:03:54.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:03:54.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:03:54.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:03:54.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:03:54.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:03:54.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:03:54.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:03:54.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:03:54.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:03:55.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:03:55.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:03:55.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:03:55.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:03:55.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:03:55.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:03:55.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:03:55.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:03:57.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:03:57.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:03:57.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:03:57.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:03:57.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:03:57.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:03:57.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:03:57.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:03:57.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:03:57.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:03:57.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:03:57.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:03:57.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:03:57.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:03:57.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:03:57.937 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:03:57.937 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:03:57.937 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:03:57.938 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:03:57.938 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:03:57.939 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:03:57.939 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:03:57.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:03:57.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:03:57.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:03:57.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:03:57.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:03:57.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:03:57.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:03:57.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:03:57.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:03:57.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:03:57.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:03:57.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:03:57.941 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:03:57.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:03:57.941 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:03:57.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:03:57.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:03:57.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:03:57.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:03:57.981 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:58.003+0000 7fc163ba3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:57.983 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:58.003+0000 7fc163ba3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:57.985 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:58.007+0000 7fc163ba3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:58.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:03:58.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:03:58.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:03:58.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:03:58.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:03:58.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:58.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:03:58.249 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:03:58.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:58.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:58.531 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:03:58.551+0000 7fc163ba3640 -1 Falling back to public interface 2024-02-05T06:03:58.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:59.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:59.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:59.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:03:59.626 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:03:59.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:59.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:59.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:00.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:00.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:00.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:04:00.940 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:04:00.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:00.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:04:01.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:01.321 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:01.343+0000 7fc163ba3640 -1 osd.1 84 log_to_monitors true 2024-02-05T06:04:02.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:02.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:02.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:04:02.254 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:04:02.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:02.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:04:02.662 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 89 up_thru 80 down_at 85 last_clean_interval [56,84) [v2:127.0.0.1:6810/1324248436,v1:127.0.0.1:6811/1324248436] [v2:127.0.0.1:6812/1324248436,v1:127.0.0.1:6813/1324248436] exists,up 170ecd0f-2bd8-43d7-a5bc-bad18d97c676 2024-02-05T06:04:02.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:04:02.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:04:02.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:04:02.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:04:02.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:04:02.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:04:02.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:04:02.663 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:04:02.663 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:04:02.664 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:04:02.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:04:02.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:04:02.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:04:02.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:04:02.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:04:02.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:04:02.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:04:02.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:04:02.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:04:03.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:04:03.084 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:04:03.084 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:04:03.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:04:03.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:04:03.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:04:03.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841548 2024-02-05T06:04:03.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841548 2024-02-05T06:04:03.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548' 2024-02-05T06:04:03.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:04:03.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:04:03.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089346 2024-02-05T06:04:03.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089346 2024-02-05T06:04:03.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548 1-382252089346' 2024-02-05T06:04:03.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:04:03.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:04:03.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383686 2024-02-05T06:04:03.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383686 2024-02-05T06:04:03.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548 1-382252089346 2-343597383686' 2024-02-05T06:04:03.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:04:03.591 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841548 2024-02-05T06:04:03.591 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:04:03.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:04:03.592 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841548 2024-02-05T06:04:03.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:04:03.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841548 2024-02-05T06:04:03.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841548' 2024-02-05T06:04:03.594 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 283467841548 2024-02-05T06:04:03.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:04:03.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841547 -lt 283467841548 2024-02-05T06:04:03.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:04:04.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:04:04.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:04:05.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841549 -lt 283467841548 2024-02-05T06:04:05.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:04:05.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089346 2024-02-05T06:04:05.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:04:05.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:04:05.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089346 2024-02-05T06:04:05.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:04:05.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089346 2024-02-05T06:04:05.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2024-02-05T06:04:05.276 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 382252089346 2024-02-05T06:04:05.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:04:05.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089346 -lt 382252089346 2024-02-05T06:04:05.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:04:05.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383686 2024-02-05T06:04:05.616 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:04:05.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:04:05.618 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383686 2024-02-05T06:04:05.618 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:04:05.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383686 2024-02-05T06:04:05.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383686' 2024-02-05T06:04:05.620 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 343597383686 2024-02-05T06:04:05.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:04:05.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383686 -lt 343597383686 2024-02-05T06:04:05.938 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:04:05.939 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:04:05.939 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:04:06.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:04:06.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:04:06.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:04:06.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:04:06.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:04:06.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:04:06.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:04:06.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:04:06.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:04:06.659 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:04:06.659 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:04:06.659 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:04:07.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T06:04:07.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-02-05T06:04:07.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:04:07.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T06:04:07.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:04:07.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:04:07.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:04:07.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:04:07.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:04:07.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:04:07.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:04:07.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:04:07.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:04:07.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:04:07.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:04:07.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:04:07.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:04:07.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:04:07.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T06:04:07.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T06:04:07.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:04:07.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:04:07.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:04:07.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:04:07.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:04:07.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:04:07.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:04:08.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:04:08.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:04:08.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:04:08.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:04:08.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:04:08.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:04:08.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:04:08.390 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:04:08.390 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:04:08.390 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:04:08.390 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:04:08.390 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:04:08.390 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:04:08.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:04:08.694 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:04:08.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:04:08.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:04:09.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:04:09.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:04:09.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:04:09.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T06:04:09.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-02-05T06:04:09.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-02-05T06:04:09.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:04:09.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:04:09.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-02-05T06:04:09.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:04:09.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:04:09.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:04:09.506 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:04:09.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:04:09.949 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:04:09.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:04:09.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:04:09.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:04:10.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-02-05T06:04:10.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-02-05T06:04:10.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T06:04:10.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-02-05T06:04:10.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:04:10.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:04:10.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:04:10.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:04:10.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-02-05T06:04:10.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:04:10.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:04:10.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:04:10.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:04:10.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:04:10.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:04:10.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:04:10.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:04:10.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:04:10.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:04:11.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:04:11.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-02-05T06:04:11.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:04:11.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:04:11.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:04:11.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:04:11.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:04:11.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-02-05T06:04:13.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:04:13.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:04:13.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:04:13.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:04:13.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:04:13.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:04:13.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:04:13.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:04:13.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:04:13.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:04:13.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:04:13.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:04:13.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:04:13.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:04:13.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:04:13.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:04:13.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:04:13.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:04:13.584 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:04:13.584 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:04:13.584 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:04:13.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:04:13.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:04:13.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:04:13.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:04:13.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:04:13.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:04:13.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:04:13.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:04:13.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:04:13.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:04:13.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:04:13.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:04:13.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:04:13.586 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:04:13.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:04:13.586 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:04:13.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:04:13.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:04:13.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:04:13.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:04:13.619 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:13.639+0000 7fdf3a8a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:13.622 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:13.643+0000 7fdf3a8a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:13.624 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:13.643+0000 7fdf3a8a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:13.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:04:13.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:04:13.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:04:13.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:04:13.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:04:13.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:13.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:04:13.900 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:04:13.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:13.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:04:14.166 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:14.187+0000 7fdf3a8a1640 -1 Falling back to public interface 2024-02-05T06:04:14.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:15.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:15.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:15.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:04:15.246 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:04:15.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:15.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:04:15.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:16.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:16.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:16.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:04:16.563 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:04:16.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:16.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:04:16.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:16.917 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:16.939+0000 7fdf3a8a1640 -1 osd.0 93 log_to_monitors true 2024-02-05T06:04:17.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:17.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:17.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:04:17.882 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:04:17.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:17.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:04:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 98 up_thru 98 down_at 94 last_clean_interval [66,93) [v2:127.0.0.1:6802/2601869516,v1:127.0.0.1:6803/2601869516] [v2:127.0.0.1:6804/2601869516,v1:127.0.0.1:6805/2601869516] exists,up c5f4bf3a-15bb-4966-baef-5caa09dc2d5b 2024-02-05T06:04:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:04:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:04:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:04:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:04:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:04:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:04:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:04:18.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:04:18.281 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:04:18.281 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:04:18.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:04:18.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:04:18.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:04:18.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:04:18.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:04:18.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:04:18.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:04:18.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:04:18.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:04:18.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:04:18.786 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:04:18.786 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:04:18.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:04:18.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:04:18.786 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:04:19.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795010 2024-02-05T06:04:19.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795010 2024-02-05T06:04:19.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010' 2024-02-05T06:04:19.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:04:19.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:04:19.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089350 2024-02-05T06:04:19.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089350 2024-02-05T06:04:19.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010 1-382252089350' 2024-02-05T06:04:19.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:04:19.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:04:19.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383690 2024-02-05T06:04:19.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383690 2024-02-05T06:04:19.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010 1-382252089350 2-343597383690' 2024-02-05T06:04:19.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:04:19.513 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-420906795010 2024-02-05T06:04:19.513 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:04:19.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:04:19.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-420906795010 2024-02-05T06:04:19.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:04:19.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795010 2024-02-05T06:04:19.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 420906795010' 2024-02-05T06:04:19.517 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 420906795010 2024-02-05T06:04:19.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:04:19.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 420906795010 2024-02-05T06:04:19.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:04:20.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:04:20.828 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:04:21.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795010 -lt 420906795010 2024-02-05T06:04:21.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:04:21.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:04:21.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089350 2024-02-05T06:04:21.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:04:21.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089350 2024-02-05T06:04:21.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:04:21.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089350 2024-02-05T06:04:21.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089350' 2024-02-05T06:04:21.140 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 382252089350 2024-02-05T06:04:21.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:04:21.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089350 -lt 382252089350 2024-02-05T06:04:21.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:04:21.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383690 2024-02-05T06:04:21.448 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:04:21.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:04:21.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383690 2024-02-05T06:04:21.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:04:21.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383690 2024-02-05T06:04:21.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383690' 2024-02-05T06:04:21.451 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 343597383690 2024-02-05T06:04:21.451 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:04:21.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383690 -lt 343597383690 2024-02-05T06:04:21.811 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:04:21.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:04:21.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:04:22.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:04:22.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:04:22.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:04:22.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:04:22.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:04:22.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:04:22.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:04:22.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:04:22.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:04:22.542 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:04:22.542 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:04:22.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:04:22.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:04:22.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:04:22.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:04:22.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-02-05T06:04:22.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T06:04:22.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:04:22.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:04:22.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:04:22.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:04:22.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T06:04:22.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:04:22.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:04:22.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:04:22.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:04:22.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:04:22.930 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:04:22.930 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:04:22.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:04:22.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:04:22.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:04:23.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:04:23.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T06:04:23.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:04:23.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:04:23.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:04:23.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:04:23.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:04:23.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T06:04:25.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:04:25.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:04:25.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:04:25.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:04:25.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:04:25.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:04:25.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:04:25.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:04:25.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:04:25.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:04:25.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:04:25.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:04:25.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:04:25.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:04:25.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:04:25.834 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:04:25.834 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:04:25.834 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:04:25.834 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:04:25.834 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:04:25.834 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:04:25.835 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:04:25.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:04:25.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:04:25.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:04:25.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:04:25.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:04:25.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:04:25.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:04:25.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:04:25.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:04:25.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:04:25.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:04:25.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:04:25.836 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:04:25.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:04:25.836 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:04:25.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:04:25.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:04:25.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:04:25.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:04:25.869 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:25.890+0000 7f28c7605640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:25.872 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:25.894+0000 7f28c7605640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:25.874 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:25.894+0000 7f28c7605640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:26.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:04:26.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:04:26.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:04:26.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:04:26.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:04:26.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:26.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:04:26.150 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:04:26.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:04:26.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:26.411 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:26.430+0000 7f28c7605640 -1 Falling back to public interface 2024-02-05T06:04:26.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:27.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:27.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:27.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:04:27.512 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:04:27.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:27.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:04:27.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:28.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:28.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:28.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:04:28.843 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:04:28.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:28.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:04:29.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:29.211 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:29.230+0000 7f28c7605640 -1 osd.1 101 log_to_monitors true 2024-02-05T06:04:30.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:30.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:30.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:04:30.156 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:04:30.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:30.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:04:30.532 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 106 up_thru 106 down_at 102 last_clean_interval [89,101) [v2:127.0.0.1:6810/1250911680,v1:127.0.0.1:6811/1250911680] [v2:127.0.0.1:6812/1250911680,v1:127.0.0.1:6813/1250911680] exists,up 170ecd0f-2bd8-43d7-a5bc-bad18d97c676 2024-02-05T06:04:30.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:04:30.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:04:30.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:04:30.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:04:30.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:04:30.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:04:30.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:04:30.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:04:30.533 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:04:30.533 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:04:30.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:04:30.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:04:30.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:04:30.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:04:30.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:04:30.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:04:30.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:04:30.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:04:30.654 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:04:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:04:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:04:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:04:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:04:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:04:30.989 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:04:31.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795013 2024-02-05T06:04:31.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795013 2024-02-05T06:04:31.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013' 2024-02-05T06:04:31.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:04:31.313 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:04:31.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533378 2024-02-05T06:04:31.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533378 2024-02-05T06:04:31.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013 1-455266533378' 2024-02-05T06:04:31.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:04:31.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:04:31.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383693 2024-02-05T06:04:31.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383693 2024-02-05T06:04:31.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013 1-455266533378 2-343597383693' 2024-02-05T06:04:31.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:04:31.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-420906795013 2024-02-05T06:04:31.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:04:31.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:04:31.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-420906795013 2024-02-05T06:04:31.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:04:31.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795013 2024-02-05T06:04:31.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 420906795013' 2024-02-05T06:04:31.723 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 420906795013 2024-02-05T06:04:31.723 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:04:32.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795012 -lt 420906795013 2024-02-05T06:04:32.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:04:33.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:04:33.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:04:33.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795014 -lt 420906795013 2024-02-05T06:04:33.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:04:33.402 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533378 2024-02-05T06:04:33.403 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:04:33.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:04:33.406 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533378 2024-02-05T06:04:33.406 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:04:33.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533378 2024-02-05T06:04:33.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533378' 2024-02-05T06:04:33.407 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 455266533378 2024-02-05T06:04:33.408 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:04:33.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533378 -lt 455266533378 2024-02-05T06:04:33.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:04:33.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383693 2024-02-05T06:04:33.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:04:33.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:04:33.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383693 2024-02-05T06:04:33.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:04:33.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383693 2024-02-05T06:04:33.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383693' 2024-02-05T06:04:33.750 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 343597383693 2024-02-05T06:04:33.750 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:04:34.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383693 -lt 343597383693 2024-02-05T06:04:34.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:04:34.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:04:34.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:04:34.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:04:34.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:04:34.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:04:34.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:04:34.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:04:34.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:04:34.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:04:34.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:04:34.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:04:34.869 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:04:34.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:04:34.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:04:35.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:04:35.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:04:35.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:04:35.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T06:04:35.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-02-05T06:04:35.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-02-05T06:04:35.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:04:35.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:04:35.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-02-05T06:04:35.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:04:35.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:04:35.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:04:35.723 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:04:35.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:04:36.234 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:04:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:04:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:04:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:04:36.456 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-02-05T06:04:36.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T06:04:36.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T06:04:36.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-02-05T06:04:36.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-02-05T06:04:36.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-02-05T06:04:36.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-02-05T06:04:36.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:04:36.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:04:36.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:04:36.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:04:36.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-02-05T06:04:36.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:04:36.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:04:36.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:04:36.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:04:36.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:04:36.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:04:36.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:04:36.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:04:36.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:04:36.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:04:36.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:04:36.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-02-05T06:04:36.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:04:36.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:04:36.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:04:36.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:04:36.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:04:36.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-02-05T06:04:39.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:04:39.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:04:39.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:04:39.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:04:39.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:04:39.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:04:39.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:04:39.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:04:39.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:04:39.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:04:39.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:04:39.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:04:39.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:04:39.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:04:39.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:04:39.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:04:39.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:04:39.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:04:39.582 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:04:39.582 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:04:39.582 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:04:39.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:04:39.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:04:39.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:04:39.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:04:39.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:04:39.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:04:39.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:04:39.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:04:39.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:04:39.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:04:39.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:04:39.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:04:39.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:04:39.584 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:04:39.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:04:39.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:04:39.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:04:39.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:04:39.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:04:39.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:04:39.618 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:39.638+0000 7fb0105cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:39.620 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:39.642+0000 7fb0105cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:39.622 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:39.642+0000 7fb0105cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:39.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:04:39.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:04:39.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:04:39.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:04:39.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:04:39.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:39.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:04:39.899 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:04:39.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:39.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:04:40.162 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:40.182+0000 7fb0105cd640 -1 Falling back to public interface 2024-02-05T06:04:40.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:41.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:41.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:41.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:04:41.211 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:04:41.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:41.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:04:41.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:42.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:42.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:42.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:04:42.520 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:04:42.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:42.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:04:42.815 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:42.834+0000 7fb0105cd640 -1 osd.0 111 log_to_monitors true 2024-02-05T06:04:42.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:43.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:43.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:43.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:04:43.847 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:04:43.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:43.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:04:44.335 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 116 up_thru 116 down_at 112 last_clean_interval [98,111) [v2:127.0.0.1:6802/2473586767,v1:127.0.0.1:6803/2473586767] [v2:127.0.0.1:6804/2473586767,v1:127.0.0.1:6805/2473586767] exists,up c5f4bf3a-15bb-4966-baef-5caa09dc2d5b 2024-02-05T06:04:44.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:04:44.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:04:44.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:04:44.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:04:44.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:04:44.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:04:44.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:04:44.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:04:44.336 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:04:44.336 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:04:44.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:04:44.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:04:44.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:04:44.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:04:44.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:04:44.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:04:44.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:04:44.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:04:44.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:04:44.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:04:44.957 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:04:44.957 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:04:44.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:04:44.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:04:44.958 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:04:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-02-05T06:04:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-02-05T06:04:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338' 2024-02-05T06:04:45.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:04:45.229 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:04:45.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533382 2024-02-05T06:04:45.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533382 2024-02-05T06:04:45.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338 1-455266533382' 2024-02-05T06:04:45.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:04:45.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:04:45.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383697 2024-02-05T06:04:45.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383697 2024-02-05T06:04:45.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338 1-455266533382 2-343597383697' 2024-02-05T06:04:45.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:04:45.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-498216206338 2024-02-05T06:04:45.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:04:45.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:04:45.548 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-498216206338 2024-02-05T06:04:45.548 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:04:45.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-02-05T06:04:45.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 498216206338' 2024-02-05T06:04:45.550 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 498216206338 2024-02-05T06:04:45.550 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:04:45.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 498216206338 2024-02-05T06:04:45.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:04:46.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:04:46.854 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:04:47.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206338 -lt 498216206338 2024-02-05T06:04:47.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:04:47.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:04:47.214 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533382 2024-02-05T06:04:47.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:04:47.215 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533382 2024-02-05T06:04:47.215 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:04:47.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533382 2024-02-05T06:04:47.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533382' 2024-02-05T06:04:47.217 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 455266533382 2024-02-05T06:04:47.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:04:47.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533382 -lt 455266533382 2024-02-05T06:04:47.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:04:47.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383697 2024-02-05T06:04:47.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:04:47.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:04:47.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383697 2024-02-05T06:04:47.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:04:47.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383697 2024-02-05T06:04:47.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383697' 2024-02-05T06:04:47.515 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 343597383697 2024-02-05T06:04:47.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:04:47.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383697 -lt 343597383697 2024-02-05T06:04:47.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:04:47.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:04:47.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:04:48.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:04:48.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:04:48.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:04:48.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:04:48.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:04:48.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:04:48.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:04:48.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:04:48.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T06:04:48.565 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:04:48.565 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:04:48.565 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:04:48.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-02-05T06:04:48.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-02-05T06:04:48.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:04:48.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-02-05T06:04:48.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:04:48.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:04:49.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:04:49.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:04:49.040 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:04:49.040 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:04:49.040 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:04:49.040 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:04:49.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:04:49.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:04:49.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T06:04:49.400 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:04:49.400 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:04:49.400 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:04:49.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-02-05T06:04:49.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-02-05T06:04:49.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:04:49.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:04:49.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:04:49.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:04:49.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:04:49.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:04:49.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:04:50.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:04:50.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:04:50.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:04:50.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:04:50.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:04:50.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:04:50.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:04:50.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:04:50.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:04:50.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:04:50.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:04:50.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:04:50.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:04:50.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:04:50.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:04:50.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:04:50.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:04:50.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:04:50.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:04:50.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:04:50.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T06:04:50.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:04:50.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:04:50.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:04:50.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:04:50.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T06:04:50.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:04:50.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:04:50.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:04:50.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:04:50.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:04:50.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:04:50.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:04:50.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:04:50.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:04:50.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:04:52.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:04:52.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T06:04:52.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:04:52.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:04:52.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:04:52.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:04:52.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:04:52.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T06:04:54.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:04:54.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:04:54.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:04:54.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:04:54.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:04:54.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:04:54.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:04:54.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:04:54.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:04:54.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:04:54.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:04:54.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:04:54.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:04:54.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:04:54.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:04:54.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:04:54.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:04:54.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:04:54.554 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:04:54.554 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:04:54.554 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:04:54.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:04:54.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:04:54.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:04:54.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:04:54.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:04:54.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:04:54.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:04:54.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:04:54.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:04:54.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:04:54.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:04:54.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:04:54.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:04:54.557 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:04:54.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:04:54.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:04:54.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:04:54.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:04:54.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:04:54.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:04:54.593 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:54.618+0000 7f64bb8b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:54.631 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:54.654+0000 7f64bb8b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:54.632 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:54.658+0000 7f64bb8b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:54.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:04:54.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:04:54.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:04:54.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:04:54.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:04:54.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:54.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:04:54.875 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:04:54.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:54.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:04:55.166 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:55.190+0000 7f64bb8b0640 -1 Falling back to public interface 2024-02-05T06:04:55.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:56.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:56.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:56.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:04:56.250 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:04:56.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:56.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:04:56.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:57.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:57.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:57.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:04:57.570 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:04:57.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:57.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:04:57.759 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:04:57.782+0000 7f64bb8b0640 -1 osd.0 118 log_to_monitors true 2024-02-05T06:04:57.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:58.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:58.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:58.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:04:58.897 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:04:58.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:58.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:04:59.236 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 122 up_thru 122 down_at 119 last_clean_interval [116,118) [v2:127.0.0.1:6802/2309816820,v1:127.0.0.1:6803/2309816820] [v2:127.0.0.1:6804/2309816820,v1:127.0.0.1:6805/2309816820] exists,up c5f4bf3a-15bb-4966-baef-5caa09dc2d5b 2024-02-05T06:04:59.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:04:59.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:04:59.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:04:59.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:04:59.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:04:59.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:04:59.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:04:59.238 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:04:59.239 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:04:59.239 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:04:59.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:04:59.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:04:59.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:04:59.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:04:59.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:04:59.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:04:59.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:04:59.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:04:59.352 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:04:59.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:04:59.782 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:04:59.782 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:04:59.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:04:59.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:04:59.783 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:05:00.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010114 2024-02-05T06:05:00.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010114 2024-02-05T06:05:00.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114' 2024-02-05T06:05:00.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:00.002 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:05:00.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533386 2024-02-05T06:05:00.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533386 2024-02-05T06:05:00.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-455266533386' 2024-02-05T06:05:00.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:00.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:05:00.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383701 2024-02-05T06:05:00.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383701 2024-02-05T06:05:00.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-455266533386 2-343597383701' 2024-02-05T06:05:00.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:00.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-523986010114 2024-02-05T06:05:00.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:00.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:05:00.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-523986010114 2024-02-05T06:05:00.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:00.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=523986010114 2024-02-05T06:05:00.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 523986010114' 2024-02-05T06:05:00.353 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 523986010114 2024-02-05T06:05:00.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:05:00.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 523986010114 2024-02-05T06:05:00.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:05:01.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:05:01.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:05:01.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010114 -lt 523986010114 2024-02-05T06:05:01.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:01.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533386 2024-02-05T06:05:01.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:01.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:05:01.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533386 2024-02-05T06:05:01.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:01.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533386 2024-02-05T06:05:01.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533386' 2024-02-05T06:05:01.987 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 455266533386 2024-02-05T06:05:01.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:05:02.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533386 -lt 455266533386 2024-02-05T06:05:02.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:02.301 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383701 2024-02-05T06:05:02.301 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:02.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:05:02.304 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383701 2024-02-05T06:05:02.304 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:02.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383701 2024-02-05T06:05:02.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383701' 2024-02-05T06:05:02.307 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.2 seq 343597383701 2024-02-05T06:05:02.307 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:05:02.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383701 -lt 343597383701 2024-02-05T06:05:02.665 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:05:02.665 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:05:02.666 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:05:03.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:05:03.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:05:03.029 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:05:03.029 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:05:03.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:05:03.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:05:03.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:05:03.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:05:03.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:05:03.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:05:03.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:05:03.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:05:03.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T06:05:03.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T06:05:03.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:05:03.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T06:05:03.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:05:03.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:05:03.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:05:03.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:05:03.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:05:03.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:05:03.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:05:03.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:05:03.833 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:05:03.833 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:05:04.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:05:04.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:05:04.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:05:04.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:05:04.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T06:05:04.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T06:05:04.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:05:04.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:05:04.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:05:04.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:05:04.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:05:04.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:05:04.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:05:04.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:05:04.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:05:04.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:05:04.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:05:04.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:05:05.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:05:05.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:05:05.099 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:05:05.099 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:05:05.099 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:05:05.099 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:05:05.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:05:05.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:05:05.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:05:05.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:05:05.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:05:05.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:05:05.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T06:05:05.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T06:05:05.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:05:05.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:05:05.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:05:05.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:05:05.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:05:05.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:05:05.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:05:06.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:05:06.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:05:06.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T06:05:06.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:05:06.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T06:05:06.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:05:06.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:05:06.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:05:06.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:05:06.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:05:06.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:05:06.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:05:06.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:05:06.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:05:06.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:05:06.849 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:05:06.849 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:05:07.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:05:07.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:05:07.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:05:07.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-02-05T06:05:07.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-02-05T06:05:07.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T06:05:07.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-02-05T06:05:07.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-02-05T06:05:07.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:05:07.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-02-05T06:05:07.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-02-05T06:05:07.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-02-05T06:05:07.571 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-02-05T06:05:07.571 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:05:07.571 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:05:07.571 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:05:07.571 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:05:07.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-02-05T06:02:19.449817+0000 2024-02-05T06:05:07.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-02-05T06:05:08.140 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-02-05T06:05:08.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-02-05T06:02:19.449817+0000 2024-02-05T06:05:08.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T06:05:08.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:02:19.449817+0000 2024-02-05T06:05:08.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:05:08.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:05:08.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:05:08.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:05:08.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:05:08.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:05:08.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:05:08.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:05:08.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:02:19.449817+0000 '>' 2024-02-05T06:02:19.449817+0000 2024-02-05T06:05:08.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:05:09.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:05:09.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:05:09.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:05:09.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:05:09.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:05:09.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:05:09.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:05:09.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:02:19.449817+0000 '>' 2024-02-05T06:02:19.449817+0000 2024-02-05T06:05:09.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:05:10.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:05:10.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:05:10.828 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:05:10.828 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:05:10.828 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:05:10.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:05:10.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:05:11.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:05:08.372463+0000 '>' 2024-02-05T06:02:19.449817+0000 2024-02-05T06:05:11.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:05:11.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-02-05T06:05:11.209 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-02-05T06:05:11.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-02-05T06:05:11.238 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-02-05T06:05:11.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-02-05T06:05:11.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-02-05T06:05:11.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-02-05T06:05:11.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=122 2024-02-05T06:05:11.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T06:05:11.322 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T06:05:11.322 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T06:05:11.322 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T06:05:11.322 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T06:05:11.322 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T06:05:11.322 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T06:05:11.322 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T06:05:11.322 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T06:05:11.322 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T06:05:11.323 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T06:05:11.323 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T06:05:11.323 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T06:05:11.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T06:05:11.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T06:05:11.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T06:05:11.362 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T06:05:11.362 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T06:05:11.362 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T06:05:11.362 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T06:05:11.362 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T06:05:11.363 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T06:05:11.363 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T06:05:11.363 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T06:05:11.363 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T06:05:11.363 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T06:05:11.363 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T06:05:11.363 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-02-05T06:05:11.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T06:05:11.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T06:05:11.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T06:05:11.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T06:05:11.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-02-05T06:05:11.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-02-05T06:05:11.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-02-05T06:05:11.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-02-05T06:05:11.403 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:05:11.403 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:05:11.403 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:05:11.403 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:05:11.403 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:05:11.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-02-05T06:02:19.449817+0000 2024-02-05T06:05:11.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-02-05T06:05:11.992 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-02-05T06:05:12.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-02-05T06:02:19.449817+0000 last_deep_scrub_stamp 2024-02-05T06:05:12.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T06:05:12.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:02:19.449817+0000 2024-02-05T06:05:12.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-02-05T06:05:12.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:05:12.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:05:12.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:05:12.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:05:12.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:05:12.012 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:05:12.012 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:05:12.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:02:19.449817+0000 '>' 2024-02-05T06:02:19.449817+0000 2024-02-05T06:05:12.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:05:13.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:05:13.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:05:13.306 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:05:13.306 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:05:13.306 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:05:13.306 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:05:13.306 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:05:13.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:02:19.449817+0000 '>' 2024-02-05T06:02:19.449817+0000 2024-02-05T06:05:13.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:05:14.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:05:14.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:05:14.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:05:14.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:05:14.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:05:14.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:05:14.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:05:14.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:02:19.449817+0000 '>' 2024-02-05T06:02:19.449817+0000 2024-02-05T06:05:14.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:05:15.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:05:15.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:05:15.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:05:15.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:05:15.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:05:15.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:05:15.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:05:16.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:02:19.449817+0000 '>' 2024-02-05T06:02:19.449817+0000 2024-02-05T06:05:16.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:05:17.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:05:17.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:05:17.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:05:17.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:05:17.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:05:17.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:05:17.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:05:17.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:05:12.532946+0000 '>' 2024-02-05T06:02:19.449817+0000 2024-02-05T06:05:17.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:05:17.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-02-05T06:05:17.607 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-02-05T06:05:17.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-02-05T06:05:17.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-02-05T06:05:17.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-02-05T06:05:17.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-02-05T06:05:17.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-02-05T06:05:17.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=122 2024-02-05T06:05:17.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-02-05T06:05:17.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T06:05:17.727 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T06:05:17.727 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T06:05:17.727 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T06:05:17.727 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T06:05:17.727 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T06:05:17.728 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T06:05:17.728 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T06:05:17.728 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T06:05:17.728 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T06:05:17.728 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T06:05:17.728 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T06:05:17.728 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T06:05:17.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T06:05:17.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T06:05:17.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T06:05:17.771 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T06:05:17.771 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T06:05:17.771 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T06:05:17.771 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T06:05:17.771 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T06:05:17.771 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T06:05:17.771 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T06:05:17.771 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T06:05:17.772 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T06:05:17.772 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T06:05:17.772 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T06:05:17.772 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-02-05T06:05:17.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T06:05:17.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T06:05:17.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T06:05:17.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T06:05:17.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-02-05T06:05:17.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-02-05T06:05:17.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-02-05T06:05:18.214 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'ecpool' removed 2024-02-05T06:05:18.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:05:18.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:05:18.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:05:18.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:05:18.235 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:05:18.235 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:05:18.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:05:18.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:05:18.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:05:18.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:05:18.347 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:05:18.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:05:18.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:05:18.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:05:18.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:05:18.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:05:18.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:05:18.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:05:18.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:05:18.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:05:18.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:05:18.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:05:18.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:05:18.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:05:18.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:18.388 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:05:18.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:05:18.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:05:18.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:05:18.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:05:18.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:05:18.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:05:18.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:05:18.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:05:18.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:05:18.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:05:18.390 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:05:18.390 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:05:18.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:05:18.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:05:18.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:05:18.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:05:18.394 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:05:18.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:05:18.394 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:05:18.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:05:18.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:05:18.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:05:18.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:05:18.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:05:18.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:05:18.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:05:18.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:05:18.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:05:18.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:05:18.399 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:05:18.399 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:18.399 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:05:18.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:05:18.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:05:18.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:05:18.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:05:18.400 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:05:18.400 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:18.400 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:05:18.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T06:05:18.404 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:05:18.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:05:18.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:05:18.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:05:18.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-02-05T06:05:18.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-02-05T06:05:18.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-02-05T06:05:18.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-02-05T06:05:18.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-02-05T06:05:18.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:05:18.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:05:18.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:05:18.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:05:18.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:05:18.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-02-05T06:05:18.799 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:05:18.799 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:05:18.800 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:05:18.800 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:05:18.800 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:18.800 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:05:18.801 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:05:18.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T06:05:19.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:05:19.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:05:19.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:05:19.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:05:19.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:05:19.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:05:19.011 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:05:19.011 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:05:19.011 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:05:19.012 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:05:19.012 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:19.012 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:05:19.012 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:05:19.013 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:05:19.013 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T06:05:19.128 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:05:19.128 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:05:19.128 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:05:19.128 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:05:19.129 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:05:19.129 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:05:19.129 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:05:19.129 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:05:19.129 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:05:19.130 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:19.130 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:05:19.130 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:05:19.130 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:05:19.130 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T06:05:19.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-02-05T06:05:19.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:05:19.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:05:19.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:05:19.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:05:19.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:05:19.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:05:20.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:05:20.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:05:20.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:05:20.124 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:05:20.124 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:20.124 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:05:20.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:05:20.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:05:20.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:05:20.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-02-05T06:05:20.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:05:20.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:05:20.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:05:20.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:05:20.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:05:20.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:05:20.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:05:20.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:05:20.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:05:20.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:05:20.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:05:20.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:05:20.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:05:20.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:05:20.167 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:05:20.167 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:05:20.167 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:05:20.167 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:05:20.167 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:20.167 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:05:20.167 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:05:20.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:05:20.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:05:20.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:05:20.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:05:20.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:05:20.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:05:20.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:05:20.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:05:20.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:05:20.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:05:20.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:05:20.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:05:20.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:05:20.169 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:05:20.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fbea8a79-eb9c-42b5-b2ac-96fd14b4911d 2024-02-05T06:05:20.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fbea8a79-eb9c-42b5-b2ac-96fd14b4911d' 2024-02-05T06:05:20.169 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 fbea8a79-eb9c-42b5-b2ac-96fd14b4911d 2024-02-05T06:05:20.170 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:05:20.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCgesBlYCmrDBAAuzNENngBxivcBEVg6+96gA== 2024-02-05T06:05:20.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCgesBlYCmrDBAAuzNENngBxivcBEVg6+96gA=="}' 2024-02-05T06:05:20.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fbea8a79-eb9c-42b5-b2ac-96fd14b4911d -i td/osd-scrub-repair/0/new.json 2024-02-05T06:05:20.448 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:05:20.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:05:20.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgesBlYCmrDBAAuzNENngBxivcBEVg6+96gA== --osd-uuid fbea8a79-eb9c-42b5-b2ac-96fd14b4911d 2024-02-05T06:05:20.498 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:20.521+0000 7f84f60dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:20.515 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:20.537+0000 7f84f60dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:20.516 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:20.541+0000 7f84f60dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:20.517 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:20.541+0000 7f84f60dd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:05:20.517 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:20.541+0000 7f84f60dd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:05:20.517 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:20.541+0000 7f84f60dd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:05:20.607 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:20.629+0000 7f84f60dd640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:05:26.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:05:26.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:05:26.793 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:05:26.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:05:26.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:05:27.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:05:27.182 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:05:27.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:05:27.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:05:27.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:05:27.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:05:27.215 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:27.237+0000 7fd776c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:27.248 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:27.273+0000 7fd776c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:27.250 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:27.273+0000 7fd776c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:27.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:05:27.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:05:27.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:05:27.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:05:27.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:05:27.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:27.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:05:27.518 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:05:27.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:27.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:27.809 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:27.833+0000 7fd776c15640 -1 Falling back to public interface 2024-02-05T06:05:27.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:28.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:28.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:28.852 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:05:28.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:05:28.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:28.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:29.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:30.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:30.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:30.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:05:30.168 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:05:30.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:30.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:30.375 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:30.397+0000 7fd776c15640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:05:30.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:31.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:31.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:31.488 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:05:31.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:05:31.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:31.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:31.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:32.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:32.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:32.805 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:05:32.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:05:32.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:32.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:33.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:05:34.266 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:05:34.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:34.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:34.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:35.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:35.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:35.594 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:05:35.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:05:35.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:35.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:35.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:36.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:36.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:36.908 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:05:36.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:05:36.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:36.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:37.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:38.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:38.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:38.227 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:05:38.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:05:38.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:38.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:38.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:39.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:39.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:39.549 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:05:39.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:05:39.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:39.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:39.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:40.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:40.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:40.870 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:05:40.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:05:40.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:40.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:41.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:42.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:42.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:42.285 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:05:42.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:05:42.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:42.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:42.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:43.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:43.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:43.604 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:05:43.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:05:43.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:43.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:43.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:44.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:44.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:44.929 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:05:44.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:05:44.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:44.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:45.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:46.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:46.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:46.596 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:05:46.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:05:46.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:46.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:47.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:48.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:48.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:48.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:05:48.431 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:05:48.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:48.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:49.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:50.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:50.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:50.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:05:50.226 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:05:50.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:50.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:51.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:52.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:52.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:52.022 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:05:52.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:05:52.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:52.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:52.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:53.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:53.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:53.729 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:05:53.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:05:53.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:53.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:54.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:55.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:55.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:55.428 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:05:55.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:05:55.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:55.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:55.738 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3925585613,v1:127.0.0.1:6803/3925585613] [v2:127.0.0.1:6804/3925585613,v1:127.0.0.1:6805/3925585613] exists,up fbea8a79-eb9c-42b5-b2ac-96fd14b4911d 2024-02-05T06:05:55.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:05:55.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:05:55.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:05:55.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-02-05T06:05:55.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:05:55.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:05:55.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:05:55.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:05:55.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:05:55.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:05:55.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:05:55.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:05:55.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:05:55.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:05:55.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:05:55.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:05:55.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:05:55.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:05:55.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:05:55.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:05:55.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:05:55.743 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:05:55.743 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:55.743 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:05:55.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:05:55.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:05:55.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:05:55.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:05:55.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:05:55.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:05:55.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:05:55.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:05:55.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:05:55.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:05:55.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:05:55.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:05:55.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:05:55.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:05:55.744 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:05:55.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:05:55.745 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:05:55.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 edd95e57-be98-4562-8431-9f2ffce6a753' 2024-02-05T06:05:55.745 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:05:55.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDesBl4JU5LxAAig+rphJ9u6asb6I2lXNlWw== 2024-02-05T06:05:55.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDesBl4JU5LxAAig+rphJ9u6asb6I2lXNlWw=="}' 2024-02-05T06:05:55.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new edd95e57-be98-4562-8431-9f2ffce6a753 -i td/osd-scrub-repair/1/new.json 2024-02-05T06:05:56.142 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:05:56.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T06:05:56.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDesBl4JU5LxAAig+rphJ9u6asb6I2lXNlWw== --osd-uuid edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:05:56.196 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:56.220+0000 7fbb51b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:56.235 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:56.260+0000 7fbb51b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:56.237 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:56.260+0000 7fbb51b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:56.237 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:56.260+0000 7fbb51b2a640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:05:56.238 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:56.260+0000 7fbb51b2a640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:05:56.238 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:56.260+0000 7fbb51b2a640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:05:56.315 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:05:56.336+0000 7fbb51b2a640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T06:06:03.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T06:06:03.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:06:03.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:06:03.316 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:06:03.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:06:03.702 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:06:03.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:06:03.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:06:03.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:06:03.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:06:03.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:06:03.736 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:06:03.760+0000 7fb046bdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:03.766 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:06:03.788+0000 7fb046bdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:03.767 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:06:03.792+0000 7fb046bdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:04.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:06:04.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:06:04.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:06:04.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:06:04.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:06:04.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:04.039 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:06:04.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:06:04.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:04.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:04.329 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:06:04.352+0000 7fb046bdb640 -1 Falling back to public interface 2024-02-05T06:06:04.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:05.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:05.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:05.381 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:06:05.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:06:05.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:05.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:05.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:06.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:06.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:06.704 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:06:06.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:06:06.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:06.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:07.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:07.027 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:06:07.052+0000 7fb046bdb640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:06:08.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:08.019 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:06:08.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:08.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:06:08.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:08.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:08.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:09.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:09.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:09.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:06:09.396 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:06:09.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:09.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:09.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:10.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:10.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:10.801 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:06:10.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:06:10.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:10.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:11.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:12.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:12.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:12.125 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:06:12.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:06:12.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:12.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:12.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:13.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:13.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:13.434 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:06:13.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:06:13.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:13.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:13.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:14.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:14.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:14.767 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:06:14.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:06:14.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:14.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:15.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:16.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:16.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:16.193 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:06:16.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:06:16.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:16.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:16.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:17.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:17.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:17.508 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:06:17.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:06:17.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:17.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:17.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:18.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:18.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:18.883 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:06:18.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:06:18.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:18.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:19.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:20.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:20.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:20.210 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:06:20.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:06:20.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:20.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:20.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:21.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:21.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:21.524 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:06:21.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:06:21.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:21.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:21.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:22.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:22.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:22.921 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:06:22.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:06:22.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:22.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:23.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:24.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:24.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:24.717 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:06:24.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:06:24.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:24.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:25.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:26.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:26.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:26.528 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:06:26.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:06:26.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:26.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:27.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:28.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:28.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:28.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:06:28.189 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:06:28.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:28.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:28.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:29.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:29.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:29.983 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:06:29.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:06:29.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:29.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:30.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:31.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:31.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:31.858 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:06:31.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:06:31.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:31.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:32.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:33.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:33.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:33.522 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:06:33.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:06:33.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:33.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:33.834 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3982563323,v1:127.0.0.1:6811/3982563323] [v2:127.0.0.1:6812/3982563323,v1:127.0.0.1:6813/3982563323] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:06:33.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:06:33.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:06:33.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:06:33.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-02-05T06:06:33.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T06:06:34.160 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T06:06:34.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T06:06:34.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T06:06:34.624 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T06:06:34.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:06:35.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T06:06:37.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-02-05T06:06:37.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:06:37.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:06:37.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:06:37.789 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:06:37.791 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:06:37.791 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:06:37.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:06:37.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:06:37.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:06:37.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:06:37.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:06:37.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:06:37.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:06:37.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:06:37.927 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:06:38.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:06:38.240 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:06:38.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:06:38.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:06:38.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:06:38.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-02-05T06:06:38.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-02-05T06:06:38.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-02-05T06:06:38.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:06:38.424 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:06:38.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-02-05T06:06:38.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-02-05T06:06:38.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672968' 2024-02-05T06:06:38.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:06:38.569 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-02-05T06:06:38.569 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:06:38.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:06:38.571 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-02-05T06:06:38.571 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:06:38.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-02-05T06:06:38.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-02-05T06:06:38.573 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836496 2024-02-05T06:06:38.573 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:06:38.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-02-05T06:06:38.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:06:38.880 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:06:38.880 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-02-05T06:06:38.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:06:38.883 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-02-05T06:06:38.883 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:06:38.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-02-05T06:06:38.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-02-05T06:06:38.885 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672968 2024-02-05T06:06:38.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:06:39.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672968 2024-02-05T06:06:39.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:06:40.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:06:40.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:06:40.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672968 2024-02-05T06:06:40.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:06:41.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:06:41.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:06:41.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672968 2024-02-05T06:06:41.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:06:41.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:06:41.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:06:42.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T06:06:42.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:06:42.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:06:42.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:06:42.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:06:42.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:06:42.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:06:42.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:06:42.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:06:42.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:06:42.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:06:42.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:06:42.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T06:06:42.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:06:42.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:06:42.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-02-05T06:06:42.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-02-05T06:06:43.404 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'foo' created 2024-02-05T06:06:43.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:06:44.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-02-05T06:06:44.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-02-05T06:06:44.878 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'csr_pool' created 2024-02-05T06:06:44.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:06:45.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-02-05T06:06:45.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:06:45.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:06:45.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:06:45.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:06:45.899 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:06:45.899 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:06:45.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:06:45.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:06:45.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:06:46.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:06:46.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:06:46.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:06:46.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:06:46.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:06:46.035 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:06:46.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:06:46.448 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:06:46.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:06:46.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:06:46.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:06:46.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-02-05T06:06:46.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-02-05T06:06:46.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-02-05T06:06:46.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:06:46.665 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:06:46.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-02-05T06:06:46.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-02-05T06:06:46.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672971' 2024-02-05T06:06:46.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:06:46.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-02-05T06:06:46.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:06:46.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:06:46.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-02-05T06:06:46.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:06:46.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-02-05T06:06:46.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-02-05T06:06:46.819 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836499 2024-02-05T06:06:46.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:06:47.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-02-05T06:06:47.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:06:48.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:06:48.265 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:06:48.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-02-05T06:06:48.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:06:49.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:06:49.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:06:49.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-02-05T06:06:49.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:06:49.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-02-05T06:06:49.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:06:49.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:06:49.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-02-05T06:06:49.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:06:49.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-02-05T06:06:49.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-02-05T06:06:49.778 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672971 2024-02-05T06:06:49.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:06:50.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-02-05T06:06:50.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:06:50.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:06:50.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:06:50.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:06:50.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:06:50.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:06:50.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:06:50.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:06:50.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:06:50.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:06:50.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:06:50.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:06:50.788 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:06:50.788 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:06:50.788 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:06:51.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:06:51.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:06:51.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:06:51.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-02-05T06:06:51.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:06:51.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-02-05T06:06:51.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-02-05T06:06:51.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:06:51.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:06:51.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-02-05T06:06:51.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:06:51.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:06:51.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:06:51.568 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:06:51.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:06:52.020 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:06:52.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:06:52.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:06:52.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:06:52.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-02-05T06:06:52.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-02-05T06:06:52.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:06:52.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-02-05T06:06:52.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-02-05T06:06:52.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:06:52.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:06:52.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-02-05T06:06:52.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:06:52.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:06:52.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:06:53.007 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:06:53.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:06:53.535 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:06:53.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:06:53.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:06:53.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:06:53.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-02-05T06:06:53.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-02-05T06:06:54.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:06:54.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-02-05T06:06:54.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-02-05T06:06:54.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:06:54.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:06:54.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-02-05T06:06:54.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:06:54.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:06:54.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:06:54.479 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:06:54.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:06:54.981 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:06:54.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:06:54.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:06:54.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:06:55.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-02-05T06:06:55.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-02-05T06:06:55.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:06:55.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-02-05T06:06:55.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-02-05T06:06:55.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:06:55.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:06:55.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-02-05T06:06:55.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:06:55.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:06:55.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:06:55.935 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:06:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:06:56.378 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:06:56.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:06:56.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:06:56.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:06:56.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-02-05T06:06:56.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-02-05T06:06:57.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:06:57.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-02-05T06:06:57.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-02-05T06:06:57.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:06:57.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:06:57.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-02-05T06:06:57.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:06:57.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:06:57.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:06:57.525 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:06:57.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:06:57.960 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:06:57.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:06:57.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:06:57.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:06:58.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-02-05T06:06:58.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-02-05T06:06:58.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:06:58.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-02-05T06:06:58.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-02-05T06:06:58.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:06:58.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:06:58.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-02-05T06:06:58.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:06:58.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:06:58.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:06:59.005 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:06:59.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:06:59.574 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:06:59.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:06:59.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:06:59.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:06:59.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-02-05T06:06:59.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-02-05T06:07:00.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:00.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-02-05T06:07:00.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-02-05T06:07:00.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:07:00.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:07:00.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-02-05T06:07:00.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:07:00.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:07:00.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:07:00.503 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:07:00.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:07:00.955 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:07:00.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:07:00.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:07:00.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:07:01.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-02-05T06:07:01.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-02-05T06:07:01.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:01.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-02-05T06:07:01.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-02-05T06:07:01.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:07:01.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:07:01.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-02-05T06:07:01.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:07:01.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:07:01.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:07:01.933 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:07:01.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:07:02.527 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:07:02.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:07:02.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:07:02.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:07:02.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-02-05T06:07:02.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-02-05T06:07:03.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:03.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-02-05T06:07:03.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-02-05T06:07:03.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:07:03.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:07:03.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-02-05T06:07:03.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:07:03.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:07:03.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:07:03.556 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:07:03.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:07:03.991 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:07:04.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:07:04.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:07:04.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:07:04.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-02-05T06:07:04.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-02-05T06:07:04.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:04.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-02-05T06:07:04.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-02-05T06:07:04.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:07:04.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:07:04.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-02-05T06:07:04.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:07:04.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:07:04.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:07:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:07:04.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:07:05.589 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:07:05.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:07:05.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:07:05.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:07:05.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-02-05T06:07:06.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-02-05T06:07:06.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:06.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-02-05T06:07:06.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-02-05T06:07:06.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:07:06.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:07:06.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-02-05T06:07:06.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:07:06.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:07:06.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:07:06.718 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:07:06.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:07:07.245 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:07:07.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:07:07.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:07:07.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:07:07.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-02-05T06:07:07.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-02-05T06:07:07.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:07.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-02-05T06:07:07.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-02-05T06:07:07.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:07:07.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:07:07.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-02-05T06:07:07.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:07:07.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:07:07.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:07:08.324 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:07:08.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:07:08.952 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:07:08.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:07:08.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:07:08.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:07:09.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-02-05T06:07:09.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-02-05T06:07:09.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:09.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-02-05T06:07:09.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-02-05T06:07:09.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:07:09.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:07:09.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-02-05T06:07:09.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:07:09.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:07:09.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:07:09.963 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:07:09.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:07:10.499 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:07:10.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:07:10.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:07:10.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:07:10.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-02-05T06:07:10.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-02-05T06:07:11.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:11.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-02-05T06:07:11.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-02-05T06:07:11.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:07:11.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:07:11.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-02-05T06:07:11.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:07:11.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:07:11.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:07:11.587 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:07:11.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:07:12.030 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:07:12.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:07:12.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:07:12.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:07:12.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-02-05T06:07:12.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-02-05T06:07:12.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:12.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-02-05T06:07:12.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-02-05T06:07:12.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:07:12.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:07:12.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-02-05T06:07:12.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:07:12.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:07:12.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:07:13.083 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:07:13.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:07:13.678 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:07:13.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:07:13.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:07:13.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:07:13.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-02-05T06:07:14.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-02-05T06:07:14.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:14.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-02-05T06:07:14.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-02-05T06:07:14.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:07:14.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:07:14.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-02-05T06:07:14.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:07:14.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:07:14.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:07:14.739 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:07:14.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:07:15.266 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:07:15.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:07:15.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:07:15.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:07:15.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-02-05T06:07:15.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-02-05T06:07:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-02-05T06:07:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-02-05T06:07:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:07:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:07:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-02-05T06:07:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:07:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:07:15.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:07:16.305 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:07:16.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:07:16.909 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:07:16.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:07:16.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:07:16.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:07:17.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-02-05T06:07:17.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-02-05T06:07:17.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:17.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-02-05T06:07:17.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-02-05T06:07:17.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:07:17.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:07:17.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-02-05T06:07:17.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:07:17.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:07:17.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:07:17.945 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:07:17.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:07:18.405 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:07:18.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:07:18.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:07:18.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:07:18.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-02-05T06:07:19.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-02-05T06:07:19.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:19.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-02-05T06:07:19.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-02-05T06:07:19.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:07:19.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:07:19.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-02-05T06:07:19.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:07:19.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:07:19.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:07:19.585 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:07:19.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:07:20.029 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:07:20.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:07:20.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:07:20.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:07:20.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-02-05T06:07:20.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-02-05T06:07:20.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-02-05T06:07:20.598 INFO:tasks.workunit.client.0.smithi148.stderr:1025+0 records in 2024-02-05T06:07:20.599 INFO:tasks.workunit.client.0.smithi148.stderr:1025+0 records out 2024-02-05T06:07:20.599 INFO:tasks.workunit.client.0.smithi148.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00365563 s, 287 MB/s 2024-02-05T06:07:20.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-02-05T06:07:20.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-02-05T06:07:20.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-02-05T06:07:20.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-02-05T06:07:20.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-02-05T06:07:20.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-02-05T06:07:20.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:07:21.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-02-05T06:07:21.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-02-05T06:07:21.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-02-05T06:07:21.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-02-05T06:07:21.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-02-05T06:07:21.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:07:21.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-02-05T06:07:21.491 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-02-05T06:07:21.491 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T06:07:21.491 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T06:07:21.492 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:07:21.492 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:07:21.492 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:07:21.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.0.asok 2024-02-05T06:07:21.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-02-05T06:07:21.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.61355/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-02-05T06:07:21.587 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:07:21.587 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-02-05T06:07:21.587 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:07:21.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-02-05T06:07:21.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:07:21.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:07:21.600 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:07:21.600 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:07:21.600 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:07:21.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.1.asok 2024-02-05T06:07:21.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-02-05T06:07:21.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.61355/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-02-05T06:07:21.696 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:07:21.696 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-02-05T06:07:21.696 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:07:21.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-02-05T06:07:21.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-02-05T06:07:21.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:07:21.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:07:21.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:07:21.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:07:21.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:07:22.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-02-05T06:06:44.837244+0000 2024-02-05T06:07:22.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-02-05T06:07:22.290 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-02-05T06:07:22.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-02-05T06:06:44.837244+0000 last_deep_scrub_stamp 2024-02-05T06:07:22.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T06:07:22.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:06:44.837244+0000 2024-02-05T06:07:22.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-02-05T06:07:22.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:07:22.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:07:22.306 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:07:22.307 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:07:22.307 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:07:22.307 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:07:22.307 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:07:22.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:06:44.837244+0000 '>' 2024-02-05T06:06:44.837244+0000 2024-02-05T06:07:22.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:07:23.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:07:23.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:07:23.608 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:07:23.608 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:07:23.608 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:07:23.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:07:23.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:07:23.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:06:44.837244+0000 '>' 2024-02-05T06:06:44.837244+0000 2024-02-05T06:07:23.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:07:24.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:07:24.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:07:24.926 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:07:24.927 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:07:24.927 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:07:24.927 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:07:24.927 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:07:25.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:07:23.468440+0000 '>' 2024-02-05T06:06:44.837244+0000 2024-02-05T06:07:25.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:07:25.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-02-05T06:07:25.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:25.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-02-05T06:07:25.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-02-05T06:07:25.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T06:07:25.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:07:25.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-02-05T06:07:25.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-02-05T06:07:25.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:07:25.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:07:25.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:07:25.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:07:25.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:07:25.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:07:25.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:07:25.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:07:25.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:07:25.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:07:25.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:07:25.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:07:25.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:07:25.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:07:25.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:07:25.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:07:25.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:07:25.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:07:25.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:07:25.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:07:25.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:07:25.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:07:25.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:07:25.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:07:27.404 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-02-05T06:07:28.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:07:28.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:07:28.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:07:28.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:07:28.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:07:28.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:07:28.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:07:28.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:07:28.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:07:28.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:07:28.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:07:28.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:07:28.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:07:28.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:07:28.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:07:28.013 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:07:28.013 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:07:28.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:07:28.015 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:07:28.015 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:07:28.015 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:07:28.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:07:28.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:07:28.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:07:28.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:07:28.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:07:28.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:07:28.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:07:28.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:07:28.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:07:28.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:07:28.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:07:28.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:07:28.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:07:28.016 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:07:28.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:07:28.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:07:28.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:07:28.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:07:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:07:28.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:07:28.050 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:28.070+0000 7f74e4414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:28.053 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:28.074+0000 7f74e4414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:28.054 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:28.078+0000 7f74e4414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:28.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:07:28.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:07:28.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:07:28.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:07:28.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:07:28.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:28.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:07:28.336 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:07:28.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:28.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:07:28.589 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:28.610+0000 7f74e4414640 -1 Falling back to public interface 2024-02-05T06:07:28.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:29.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:29.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:29.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:07:29.665 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:07:29.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:29.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:07:29.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:30.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:30.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:30.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:07:30.977 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:07:30.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:30.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:07:31.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:31.426 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:31.446+0000 7f74e4414640 -1 osd.1 64 log_to_monitors true 2024-02-05T06:07:32.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:32.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:32.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:07:32.300 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:07:32.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:32.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:07:32.725 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [10,64) [v2:127.0.0.1:6810/737781799,v1:127.0.0.1:6811/737781799] [v2:127.0.0.1:6812/737781799,v1:127.0.0.1:6813/737781799] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:07:32.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:07:32.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:07:32.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:07:32.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:07:32.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:07:32.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:07:32.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:07:32.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:07:32.727 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:07:32.727 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:07:32.727 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:07:32.727 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:07:32.727 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:07:32.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:07:32.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:07:32.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:07:32.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:07:32.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:07:32.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:07:33.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:07:33.307 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:07:33.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:07:33.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:07:33.307 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:07:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-02-05T06:07:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-02-05T06:07:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-02-05T06:07:33.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:07:33.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:07:33.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776130 2024-02-05T06:07:33.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776130 2024-02-05T06:07:33.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-292057776130' 2024-02-05T06:07:33.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:07:33.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-02-05T06:07:33.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:07:33.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:07:33.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-02-05T06:07:33.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:07:33.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-02-05T06:07:33.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-02-05T06:07:33.607 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836509 2024-02-05T06:07:33.607 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:07:33.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-02-05T06:07:33.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:07:34.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:07:34.918 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:07:35.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-02-05T06:07:35.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:07:35.232 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-292057776130 2024-02-05T06:07:35.232 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:07:35.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:07:35.232 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-292057776130 2024-02-05T06:07:35.232 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:07:35.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776130 2024-02-05T06:07:35.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 292057776130' 2024-02-05T06:07:35.233 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 292057776130 2024-02-05T06:07:35.234 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:07:35.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776130 -lt 292057776130 2024-02-05T06:07:35.542 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:07:35.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:07:35.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:07:36.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:07:36.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:07:36.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:07:36.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:07:36.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:07:36.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:07:36.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:07:36.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:07:36.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:07:36.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:07:36.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:07:36.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:07:36.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:07:36.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:07:36.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:07:36.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:36.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-02-05T06:07:36.667 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-02-05T06:07:36.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T06:07:36.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:07:36.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-02-05T06:07:36.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-02-05T06:07:36.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:07:36.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:07:36.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:07:36.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:07:36.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:07:36.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:07:36.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:07:36.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:07:36.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:07:36.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:07:36.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:07:36.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:07:36.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:07:36.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:07:36.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:07:36.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:07:36.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:07:36.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:07:36.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:07:36.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:07:36.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:07:36.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:07:36.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:07:36.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:07:39.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:07:39.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:07:39.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:07:39.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:07:39.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:07:39.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:07:39.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:07:39.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:07:39.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:07:39.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:07:39.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:07:39.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:07:39.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:07:39.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:07:39.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:07:39.103 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:07:39.103 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:07:39.103 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:07:39.103 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:07:39.103 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:07:39.104 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:07:39.105 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:07:39.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:07:39.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:07:39.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:07:39.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:07:39.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:07:39.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:07:39.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:07:39.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:07:39.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:07:39.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:07:39.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:07:39.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:07:39.106 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:07:39.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:07:39.107 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:07:39.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:07:39.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:07:39.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:07:39.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:07:39.140 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:39.162+0000 7f8b98504640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:39.143 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:39.166+0000 7f8b98504640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:39.145 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:39.166+0000 7f8b98504640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:39.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:07:39.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:07:39.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:07:39.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:07:39.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:07:39.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:39.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:07:39.434 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:07:39.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:39.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:07:39.681 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:39.702+0000 7f8b98504640 -1 Falling back to public interface 2024-02-05T06:07:39.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:40.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:40.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:40.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:07:40.797 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:07:40.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:40.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:07:41.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:42.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:42.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:42.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:07:42.115 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:07:42.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:42.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:07:42.288 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:42.310+0000 7f8b98504640 -1 osd.0 70 log_to_monitors true 2024-02-05T06:07:42.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:43.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:43.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:43.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:07:43.440 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:07:43.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:43.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:07:43.763 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 74 up_thru 74 down_at 71 last_clean_interval [5,70) [v2:127.0.0.1:6802/4064243845,v1:127.0.0.1:6803/4064243845] [v2:127.0.0.1:6804/4064243845,v1:127.0.0.1:6805/4064243845] exists,up fbea8a79-eb9c-42b5-b2ac-96fd14b4911d 2024-02-05T06:07:43.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:07:43.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:07:43.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:07:43.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:07:43.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:07:43.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:07:43.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:07:43.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:07:43.765 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:07:43.765 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:07:43.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:07:43.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:07:43.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:07:43.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:07:43.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:07:43.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:07:43.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:07:43.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:07:43.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:07:44.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:07:44.189 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:07:44.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:07:44.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:07:44.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:07:44.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-02-05T06:07:44.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-02-05T06:07:44.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-317827579906' 2024-02-05T06:07:44.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:07:44.375 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:07:44.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776133 2024-02-05T06:07:44.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776133 2024-02-05T06:07:44.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-317827579906 1-292057776133' 2024-02-05T06:07:44.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:07:44.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-317827579906 2024-02-05T06:07:44.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:07:44.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:07:44.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-317827579906 2024-02-05T06:07:44.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:07:44.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-02-05T06:07:44.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 317827579906' 2024-02-05T06:07:44.523 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 317827579906 2024-02-05T06:07:44.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:07:44.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-02-05T06:07:44.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:07:44.821 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-292057776133 2024-02-05T06:07:44.821 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:07:44.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:07:44.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:07:44.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-292057776133 2024-02-05T06:07:44.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776133 2024-02-05T06:07:44.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 292057776133' 2024-02-05T06:07:44.825 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 292057776133 2024-02-05T06:07:44.826 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:07:45.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776133 -lt 292057776133 2024-02-05T06:07:45.166 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:07:45.167 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:07:45.167 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:07:45.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:07:45.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:07:45.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:07:45.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:07:45.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:07:45.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:07:45.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:07:45.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:07:45.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:07:45.948 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:07:45.948 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:07:45.948 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:07:46.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:07:46.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:07:46.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:07:46.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:46.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-02-05T06:07:46.297 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-02-05T06:07:46.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T06:07:46.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:07:46.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-02-05T06:07:46.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:07:46.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:07:46.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:07:46.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:07:46.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-02-05T06:07:46.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:07:46.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:07:46.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:07:46.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:07:46.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:07:46.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:07:46.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:07:46.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:07:46.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:07:46.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:07:46.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:07:46.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-02-05T06:07:46.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:07:46.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:07:46.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:07:46.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:07:46.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:07:46.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-02-05T06:07:48.123 INFO:tasks.workunit.client.0.smithi148.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-02-05T06:07:48.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:07:48.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:07:48.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:07:48.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:07:48.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:07:48.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:07:48.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:07:48.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:07:48.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:07:48.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:07:48.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:07:48.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:07:48.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:07:48.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:07:48.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:07:48.667 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:07:48.667 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:07:48.668 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:07:48.668 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:07:48.668 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:07:48.668 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:07:48.668 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:07:48.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:07:48.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:07:48.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:07:48.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:07:48.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:07:48.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:07:48.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:07:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:07:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:07:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:07:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:07:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:07:48.669 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:07:48.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:07:48.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:07:48.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:07:48.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:07:48.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:07:48.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:07:48.703 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:48.730+0000 7f6729581640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:48.706 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:48.734+0000 7f6729581640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:48.707 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:48.734+0000 7f6729581640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:07:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:07:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:07:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:07:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:07:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:07:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:07:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:48.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:07:49.257 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:49.282+0000 7f6729581640 -1 Falling back to public interface 2024-02-05T06:07:49.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:50.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:50.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:50.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:07:50.365 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:07:50.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:50.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:07:50.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:51.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:51.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:51.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:07:51.675 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:07:51.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:51.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:07:51.973 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:51.998+0000 7f6729581640 -1 osd.1 75 log_to_monitors true 2024-02-05T06:07:51.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:52.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:52.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:52.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:07:52.982 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:07:52.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:52.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:07:53.353 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 79 up_thru 79 down_at 76 last_clean_interval [68,75) [v2:127.0.0.1:6810/2240237142,v1:127.0.0.1:6811/2240237142] [v2:127.0.0.1:6812/2240237142,v1:127.0.0.1:6813/2240237142] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:07:53.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:07:53.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:07:53.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:07:53.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:07:53.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:07:53.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:07:53.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:07:53.355 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:07:53.355 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:07:53.355 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:07:53.355 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:07:53.355 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:07:53.355 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:07:53.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:07:53.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:07:53.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:07:53.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:07:53.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:07:53.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:07:53.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:07:53.782 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:07:53.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:07:53.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:07:53.783 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:07:53.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579909 2024-02-05T06:07:53.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579909 2024-02-05T06:07:53.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-317827579909' 2024-02-05T06:07:53.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:07:53.931 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:07:54.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416386 2024-02-05T06:07:54.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416386 2024-02-05T06:07:54.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-317827579909 1-339302416386' 2024-02-05T06:07:54.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:07:54.075 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-317827579909 2024-02-05T06:07:54.075 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:07:54.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:07:54.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-317827579909 2024-02-05T06:07:54.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:07:54.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579909 2024-02-05T06:07:54.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 317827579909' 2024-02-05T06:07:54.078 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 317827579909 2024-02-05T06:07:54.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:07:54.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579909 2024-02-05T06:07:54.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:07:55.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:07:55.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:07:55.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579909 -lt 317827579909 2024-02-05T06:07:55.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:07:55.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:07:55.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-339302416386 2024-02-05T06:07:55.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:07:55.687 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-339302416386 2024-02-05T06:07:55.687 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:07:55.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416386 2024-02-05T06:07:55.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 339302416386' 2024-02-05T06:07:55.689 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 339302416386 2024-02-05T06:07:55.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:07:56.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416386 -lt 339302416386 2024-02-05T06:07:56.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:07:56.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:07:56.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:07:56.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:07:56.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:07:56.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:07:56.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:07:56.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:07:56.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:07:56.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:07:56.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:07:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:07:56.759 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:07:56.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:07:56.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:07:57.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:07:57.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:07:57.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:07:57.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:07:57.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-02-05T06:07:57.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-02-05T06:07:57.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T06:07:57.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:07:57.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-02-05T06:07:57.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:07:57.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:07:57.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:07:57.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:07:57.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-02-05T06:07:57.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:07:57.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:07:57.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:07:57.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:07:57.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:07:57.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:07:57.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:07:57.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:07:57.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:07:57.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:07:57.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:07:57.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-02-05T06:07:57.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:07:57.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:07:57.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:07:57.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:07:57.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:07:57.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-02-05T06:07:59.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:07:59.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:07:59.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:07:59.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:07:59.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:07:59.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:07:59.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:07:59.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:07:59.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:07:59.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:07:59.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:07:59.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:07:59.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:07:59.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:07:59.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:07:59.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:07:59.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:07:59.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:07:59.506 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:07:59.506 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:07:59.506 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:07:59.507 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:07:59.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:07:59.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:07:59.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:07:59.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:07:59.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:07:59.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:07:59.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:07:59.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:07:59.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:07:59.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:07:59.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:07:59.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:07:59.508 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:07:59.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:07:59.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:07:59.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:07:59.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:07:59.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:07:59.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:07:59.542 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:59.570+0000 7fe8ca4fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:59.545 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:59.570+0000 7fe8ca4fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:59.546 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:07:59.574+0000 7fe8ca4fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:59.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:07:59.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:07:59.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:07:59.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:07:59.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:07:59.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:59.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:07:59.834 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:07:59.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:59.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:00.101 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:00.126+0000 7fe8ca4fc640 -1 Falling back to public interface 2024-02-05T06:08:00.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:01.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:01.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:01.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:08:01.174 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:08:01.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:01.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:01.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:02.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:02.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:02.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:08:02.476 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:08:02.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:02.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:02.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:02.978 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:03.006+0000 7fe8ca4fc640 -1 osd.0 80 log_to_monitors true 2024-02-05T06:08:03.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:03.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:03.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:08:03.855 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:08:03.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:03.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:04.492 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [74,80) [v2:127.0.0.1:6802/2308378328,v1:127.0.0.1:6803/2308378328] [v2:127.0.0.1:6804/2308378328,v1:127.0.0.1:6805/2308378328] exists,up fbea8a79-eb9c-42b5-b2ac-96fd14b4911d 2024-02-05T06:08:04.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:08:04.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:08:04.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:08:04.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:08:04.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:08:04.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:08:04.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:08:04.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:08:04.494 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:08:04.494 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:08:04.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:08:04.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:08:04.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:08:04.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:08:04.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:08:04.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:08:04.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:08:04.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:08:04.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:08:05.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:08:05.060 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:08:05.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:08:05.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:05.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:08:05.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-02-05T06:08:05.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-02-05T06:08:05.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252866' 2024-02-05T06:08:05.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:05.204 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:08:05.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416389 2024-02-05T06:08:05.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416389 2024-02-05T06:08:05.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252866 1-339302416389' 2024-02-05T06:08:05.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:05.347 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-360777252866 2024-02-05T06:08:05.348 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:05.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:08:05.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-360777252866 2024-02-05T06:08:05.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:05.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-02-05T06:08:05.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 360777252866' 2024-02-05T06:08:05.351 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 360777252866 2024-02-05T06:08:05.352 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:05.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 360777252866 2024-02-05T06:08:05.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:08:06.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:08:06.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:06.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252866 -lt 360777252866 2024-02-05T06:08:06.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:06.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-339302416389 2024-02-05T06:08:06.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:06.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:08:06.980 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-339302416389 2024-02-05T06:08:06.980 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:06.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416389 2024-02-05T06:08:06.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 339302416389' 2024-02-05T06:08:06.981 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 339302416389 2024-02-05T06:08:06.981 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:08:07.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416389 -lt 339302416389 2024-02-05T06:08:07.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:08:07.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:07.296 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:07.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:08:07.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:08:07.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:08:07.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:08:07.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:08:07.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:08:07.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:08:07.665 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:08:08.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:08:08.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:08:08.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:08.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:08.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:08:08.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:08:08.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:08:08.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:08:08.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-02-05T06:08:08.366 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-02-05T06:08:08.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T06:08:08.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:08:08.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-02-05T06:08:08.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:08:08.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:08:08.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:08:08.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:08:08.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-02-05T06:08:08.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:08:08.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:08:08.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:08:08.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:08:08.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:08:08.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:08:08.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:08:08.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:08:08.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:08:08.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:08:08.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:08:08.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-02-05T06:08:08.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:08:08.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:08:08.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:08:08.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:08:08.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:08:08.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-02-05T06:08:10.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:08:10.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:08:10.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:08:10.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:08:10.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:08:10.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:08:10.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:08:10.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:08:10.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:08:10.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:08:10.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:08:10.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:08:10.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:08:10.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:08:10.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:08:10.781 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:08:10.781 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:08:10.781 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:08:10.782 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:08:10.782 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:10.782 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:08:10.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:08:10.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:08:10.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:08:10.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:08:10.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:08:10.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:08:10.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:08:10.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:08:10.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:08:10.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:08:10.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:08:10.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:08:10.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:08:10.785 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:08:10.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:08:10.785 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:08:10.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:08:10.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:08:10.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:08:10.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:08:10.809 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:10.833+0000 7f84fc191640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:10.812 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:10.837+0000 7f84fc191640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:10.813 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:10.837+0000 7f84fc191640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:11.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:08:11.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:08:11.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:08:11.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:08:11.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:08:11.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:11.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:08:11.101 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:08:11.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:11.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:08:11.352 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:11.377+0000 7f84fc191640 -1 Falling back to public interface 2024-02-05T06:08:11.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:12.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:12.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:12.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:08:12.464 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:08:12.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:12.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:08:12.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:13.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:13.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:13.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:08:13.786 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:08:13.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:13.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:08:13.947 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:13.973+0000 7f84fc191640 -1 osd.1 86 log_to_monitors true 2024-02-05T06:08:14.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:15.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:15.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:15.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:08:15.113 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:08:15.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:15.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:08:15.443 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 90 up_thru 90 down_at 87 last_clean_interval [79,86) [v2:127.0.0.1:6810/2829718482,v1:127.0.0.1:6811/2829718482] [v2:127.0.0.1:6812/2829718482,v1:127.0.0.1:6813/2829718482] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:08:15.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:08:15.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:08:15.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:08:15.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:08:15.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:08:15.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:08:15.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:08:15.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:08:15.445 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:08:15.445 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:08:15.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:08:15.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:08:15.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:08:15.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:08:15.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:08:15.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:08:15.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:08:15.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:08:15.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:08:15.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:08:15.877 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:08:15.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:08:15.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:15.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:08:16.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252869 2024-02-05T06:08:16.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252869 2024-02-05T06:08:16.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252869' 2024-02-05T06:08:16.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:16.132 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:08:16.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-02-05T06:08:16.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-02-05T06:08:16.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252869 1-386547056642' 2024-02-05T06:08:16.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:16.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-360777252869 2024-02-05T06:08:16.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:16.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:08:16.285 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-360777252869 2024-02-05T06:08:16.285 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:16.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252869 2024-02-05T06:08:16.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 360777252869' 2024-02-05T06:08:16.286 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 360777252869 2024-02-05T06:08:16.286 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:16.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252868 -lt 360777252869 2024-02-05T06:08:16.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:08:17.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:08:17.597 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:17.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252869 -lt 360777252869 2024-02-05T06:08:17.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:17.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056642 2024-02-05T06:08:17.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:17.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:08:17.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056642 2024-02-05T06:08:17.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:17.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-02-05T06:08:17.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-02-05T06:08:17.914 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 386547056642 2024-02-05T06:08:17.914 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:08:18.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056642 -lt 386547056642 2024-02-05T06:08:18.230 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:08:18.231 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:18.231 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:08:18.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:08:18.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:08:18.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:08:18.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:08:18.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:08:18.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:08:18.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:08:18.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:08:18.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:08:18.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:18.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:19.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:08:19.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:08:19.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:08:19.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:08:19.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-02-05T06:08:19.371 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-02-05T06:08:19.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T06:08:19.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:08:19.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-02-05T06:08:19.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-02-05T06:08:19.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:08:19.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:08:19.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:08:19.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:08:19.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-02-05T06:08:19.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:08:19.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:08:19.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:08:19.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:08:19.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:08:19.374 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:08:19.374 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:08:19.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:08:19.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:08:19.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:08:19.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:08:19.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-02-05T06:08:19.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:08:19.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:08:19.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:08:19.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:08:19.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:08:19.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-02-05T06:08:21.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:08:21.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:08:21.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:08:21.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:08:21.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:08:21.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:08:21.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:08:21.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:08:21.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:08:21.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:08:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:08:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:08:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:08:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:08:21.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:08:21.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:08:21.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:08:21.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:08:21.651 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:08:21.651 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:21.651 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:08:21.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:08:21.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:08:21.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:08:21.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:08:21.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:08:21.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:08:21.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:08:21.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:08:21.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:08:21.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:08:21.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:08:21.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:08:21.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:08:21.654 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:08:21.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:08:21.654 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:08:21.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:08:21.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:08:21.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:08:21.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:08:21.693 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:21.717+0000 7f178885b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:21.695 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:21.721+0000 7f178885b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:21.697 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:21.721+0000 7f178885b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:21.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:08:21.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:08:21.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:08:21.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:08:21.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:08:21.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:21.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:08:21.969 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:08:21.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:21.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:22.244 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:22.269+0000 7f178885b640 -1 Falling back to public interface 2024-02-05T06:08:22.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:23.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:23.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:23.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:08:23.313 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:08:23.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:23.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:23.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:24.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:24.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:24.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:08:24.621 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:08:24.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:24.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:24.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:25.024 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:25.049+0000 7f178885b640 -1 osd.0 91 log_to_monitors true 2024-02-05T06:08:25.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:25.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:25.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:08:25.977 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:08:25.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:25.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:26.326 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 95 up_thru 95 down_at 92 last_clean_interval [84,91) [v2:127.0.0.1:6802/2657613360,v1:127.0.0.1:6803/2657613360] [v2:127.0.0.1:6804/2657613360,v1:127.0.0.1:6805/2657613360] exists,up fbea8a79-eb9c-42b5-b2ac-96fd14b4911d 2024-02-05T06:08:26.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:08:26.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:08:26.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:08:26.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:08:26.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:08:26.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:08:26.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:08:26.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:08:26.328 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:08:26.328 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:08:26.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:08:26.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:08:26.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:08:26.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:08:26.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:08:26.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:08:26.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:08:26.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:08:26.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:08:26.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:08:26.977 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:08:26.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:08:26.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:26.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:08:27.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-02-05T06:08:27.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-02-05T06:08:27.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893122' 2024-02-05T06:08:27.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:27.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:08:27.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056645 2024-02-05T06:08:27.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056645 2024-02-05T06:08:27.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893122 1-386547056645' 2024-02-05T06:08:27.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:27.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-408021893122 2024-02-05T06:08:27.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:27.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:08:27.355 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-408021893122 2024-02-05T06:08:27.355 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:27.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-02-05T06:08:27.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 408021893122' 2024-02-05T06:08:27.356 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 408021893122 2024-02-05T06:08:27.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:27.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 408021893122 2024-02-05T06:08:27.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:08:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:08:28.653 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:28.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893122 -lt 408021893122 2024-02-05T06:08:28.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:28.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:28.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056645 2024-02-05T06:08:28.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:08:28.974 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056645 2024-02-05T06:08:28.974 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:28.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056645 2024-02-05T06:08:28.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056645' 2024-02-05T06:08:28.976 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 386547056645 2024-02-05T06:08:28.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:08:29.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056645 -lt 386547056645 2024-02-05T06:08:29.307 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:08:29.308 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:29.308 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:29.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:08:29.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:08:29.696 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:08:29.696 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:08:29.697 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:08:29.697 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:08:29.697 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:08:29.697 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:08:30.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:08:30.035 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:08:30.035 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:30.036 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:30.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:08:30.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:08:30.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:08:30.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-02-05T06:08:30.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:08:30.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-02-05T06:08:30.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-02-05T06:08:30.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T06:08:30.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:08:30.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-02-05T06:08:30.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-02-05T06:08:30.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:08:30.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:08:30.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:08:30.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:08:30.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-02-05T06:08:30.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:08:30.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:08:30.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:08:30.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:08:30.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:08:30.398 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:08:30.398 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:08:30.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:08:30.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:08:30.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:08:30.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:08:30.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-02-05T06:08:30.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:08:30.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:08:30.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:08:30.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:08:30.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:08:30.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-02-05T06:08:32.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:08:32.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:08:32.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:08:32.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:08:32.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:08:32.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:08:32.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:08:32.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:08:32.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:08:32.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:08:32.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:08:32.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:08:32.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:08:32.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:08:32.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:08:32.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:08:32.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:08:32.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:08:32.595 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:08:32.595 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:32.596 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:08:32.596 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:08:32.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:08:32.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:08:32.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:08:32.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:08:32.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:08:32.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:08:32.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:08:32.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:08:32.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:08:32.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:08:32.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:08:32.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:08:32.597 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:08:32.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:08:32.597 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:08:32.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:08:32.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:08:32.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:08:32.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:08:32.631 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:32.657+0000 7f3b3b91f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:32.634 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:32.661+0000 7f3b3b91f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:32.635 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:32.661+0000 7f3b3b91f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:32.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:08:32.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:08:32.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:08:32.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:08:32.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:08:32.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:32.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:08:32.920 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:08:32.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:32.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:08:33.167 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:33.193+0000 7f3b3b91f640 -1 Falling back to public interface 2024-02-05T06:08:33.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:34.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:34.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:34.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:08:34.298 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:08:34.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:34.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:08:34.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:35.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:35.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:35.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:08:35.608 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:08:35.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:35.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:08:35.836 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:35.861+0000 7f3b3b91f640 -1 osd.1 97 log_to_monitors true 2024-02-05T06:08:35.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:36.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:36.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:36.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:08:36.961 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:08:36.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:36.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:08:37.277 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 101 up_thru 101 down_at 98 last_clean_interval [90,97) [v2:127.0.0.1:6810/1710362608,v1:127.0.0.1:6811/1710362608] [v2:127.0.0.1:6812/1710362608,v1:127.0.0.1:6813/1710362608] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:08:37.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:08:37.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:08:37.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:08:37.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:08:37.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:08:37.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:08:37.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:08:37.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:08:37.279 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:08:37.279 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:08:37.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:08:37.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:08:37.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:08:37.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:08:37.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:08:37.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:08:37.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:08:37.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:08:37.393 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:08:37.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:08:37.697 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:08:37.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:08:37.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:37.697 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:08:37.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893125 2024-02-05T06:08:37.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893125 2024-02-05T06:08:37.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893125' 2024-02-05T06:08:37.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:37.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:08:38.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696898 2024-02-05T06:08:38.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696898 2024-02-05T06:08:38.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893125 1-433791696898' 2024-02-05T06:08:38.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:38.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-408021893125 2024-02-05T06:08:38.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:38.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:08:38.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-408021893125 2024-02-05T06:08:38.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:38.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893125 2024-02-05T06:08:38.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 408021893125' 2024-02-05T06:08:38.013 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 408021893125 2024-02-05T06:08:38.013 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:38.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893125 2024-02-05T06:08:38.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:08:39.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:08:39.307 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:39.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893125 -lt 408021893125 2024-02-05T06:08:39.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:39.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-433791696898 2024-02-05T06:08:39.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:39.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:08:39.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-433791696898 2024-02-05T06:08:39.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:39.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696898 2024-02-05T06:08:39.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 433791696898' 2024-02-05T06:08:39.627 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 433791696898 2024-02-05T06:08:39.627 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:08:39.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696898 -lt 433791696898 2024-02-05T06:08:39.947 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:08:39.948 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:39.948 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:40.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:08:40.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:08:40.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:08:40.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:08:40.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:08:40.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:08:40.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:08:40.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:08:40.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:08:40.663 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:08:40.663 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:40.663 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:41.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:08:41.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:08:41.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:08:41.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-02-05T06:08:41.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:08:41.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-02-05T06:08:41.025 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-02-05T06:08:41.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T06:08:41.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:08:41.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-02-05T06:08:41.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-02-05T06:08:41.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-02-05T06:08:41.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-02-05T06:08:41.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:08:41.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:08:41.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:08:41.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:08:41.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-02-05T06:08:41.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:08:41.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:08:41.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:08:41.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:08:41.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:08:41.318 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:08:41.318 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:08:41.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:08:41.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:08:41.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:08:42.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:08:42.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-02-05T06:08:42.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:08:42.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:08:42.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:08:42.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:08:42.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:08:42.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-02-05T06:08:44.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:08:44.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:08:44.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:08:44.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:08:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:08:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:08:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:08:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:08:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:08:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:08:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:08:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:08:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:08:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:08:44.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:08:44.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:08:44.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:08:44.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:08:44.228 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:08:44.228 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:44.228 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:08:44.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:08:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:08:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:08:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:08:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:08:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:08:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:08:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:08:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:08:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:08:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:08:44.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:08:44.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:08:44.230 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:08:44.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:08:44.231 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:08:44.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:08:44.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:08:44.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:08:44.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:08:44.264 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:44.289+0000 7f84317bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:44.267 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:44.293+0000 7f84317bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:44.268 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:44.293+0000 7f84317bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:44.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:08:44.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:08:44.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:08:44.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:08:44.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:08:44.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:44.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:08:44.558 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:08:44.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:44.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:44.832 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:44.857+0000 7f84317bd640 -1 Falling back to public interface 2024-02-05T06:08:44.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:45.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:45.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:45.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:08:45.919 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:08:45.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:45.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:46.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:47.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:47.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:47.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:08:47.236 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:08:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:47.557 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:47.581+0000 7f84317bd640 -1 osd.0 102 log_to_monitors true 2024-02-05T06:08:47.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:48.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:48.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:48.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:08:48.716 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:08:48.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:48.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:49.066 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 106 up_thru 106 down_at 103 last_clean_interval [95,102) [v2:127.0.0.1:6802/667515026,v1:127.0.0.1:6803/667515026] [v2:127.0.0.1:6804/667515026,v1:127.0.0.1:6805/667515026] exists,up fbea8a79-eb9c-42b5-b2ac-96fd14b4911d 2024-02-05T06:08:49.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:08:49.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:08:49.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:08:49.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:08:49.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:08:49.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:08:49.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:08:49.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:08:49.068 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:08:49.068 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:08:49.068 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:08:49.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:08:49.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:08:49.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:08:49.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:08:49.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:08:49.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:08:49.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:08:49.172 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:08:49.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:08:49.592 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:08:49.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:08:49.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:49.592 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:08:49.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533378 2024-02-05T06:08:49.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533378 2024-02-05T06:08:49.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533378' 2024-02-05T06:08:49.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:49.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:08:49.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696901 2024-02-05T06:08:49.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696901 2024-02-05T06:08:49.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533378 1-433791696901' 2024-02-05T06:08:49.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:49.884 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533378 2024-02-05T06:08:49.884 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:49.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:08:49.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533378 2024-02-05T06:08:49.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:49.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533378 2024-02-05T06:08:49.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533378' 2024-02-05T06:08:49.887 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 455266533378 2024-02-05T06:08:49.887 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:50.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 455266533378 2024-02-05T06:08:50.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:08:51.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:08:51.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:51.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533378 -lt 455266533378 2024-02-05T06:08:51.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:51.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-433791696901 2024-02-05T06:08:51.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:51.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:08:51.489 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:51.489 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-433791696901 2024-02-05T06:08:51.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696901 2024-02-05T06:08:51.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 433791696901' 2024-02-05T06:08:51.490 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 433791696901 2024-02-05T06:08:51.490 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:08:51.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696901 -lt 433791696901 2024-02-05T06:08:51.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:08:51.817 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:51.817 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:52.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:08:52.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:08:52.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:08:52.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:08:52.211 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:08:52.211 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:08:52.211 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:08:52.211 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:08:52.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:08:52.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:08:52.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:52.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:52.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:08:52.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:08:52.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:08:52.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-02-05T06:08:52.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:08:52.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:08:52.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:08:52.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:08:52.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-02-05T06:08:52.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:08:52.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:08:52.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:08:52.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:08:52.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:08:52.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:08:52.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:08:52.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:08:52.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:08:52.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:08:53.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:08:53.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-02-05T06:08:53.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:08:53.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:08:53.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:08:53.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:08:53.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:08:53.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-02-05T06:08:55.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:08:55.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:08:55.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:08:55.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:08:55.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:08:55.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:08:55.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:08:55.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:08:55.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:08:55.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:08:55.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:08:55.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:08:55.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:08:55.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:08:55.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:08:55.311 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:08:55.312 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:08:55.312 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:08:55.312 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:08:55.312 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:55.312 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:08:55.313 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:08:55.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:08:55.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:08:55.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:08:55.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:08:55.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:08:55.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:08:55.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:08:55.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:08:55.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:08:55.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:08:55.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:08:55.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:08:55.317 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:08:55.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:08:55.317 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:08:55.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:08:55.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:08:55.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:08:55.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:08:55.343 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:55.368+0000 7f47f4c67640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:55.346 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:55.372+0000 7f47f4c67640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:55.347 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:55.372+0000 7f47f4c67640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:55.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:08:55.887 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:55.912+0000 7f47f4c67640 -1 Falling back to public interface 2024-02-05T06:08:55.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:56.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:56.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:56.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:08:56.982 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:08:56.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:56.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:08:57.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:58.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:58.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:58.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:08:58.305 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:08:58.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:58.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:08:58.566 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:08:58.592+0000 7f47f4c67640 -1 osd.1 108 log_to_monitors true 2024-02-05T06:08:58.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:59.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:59.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:59.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:08:59.670 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:08:59.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:59.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:00.151 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 112 up_thru 112 down_at 109 last_clean_interval [101,108) [v2:127.0.0.1:6810/3814725619,v1:127.0.0.1:6811/3814725619] [v2:127.0.0.1:6812/3814725619,v1:127.0.0.1:6813/3814725619] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:09:00.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:09:00.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:09:00.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:09:00.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:09:00.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:09:00.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:09:00.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:09:00.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:09:00.153 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:09:00.153 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:09:00.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:09:00.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:09:00.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:09:00.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:09:00.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:09:00.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:09:00.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:09:00.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:09:00.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:09:00.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:09:00.578 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:09:00.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:09:00.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:09:00.578 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:09:00.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533381 2024-02-05T06:09:00.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533381 2024-02-05T06:09:00.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533381' 2024-02-05T06:09:00.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:09:00.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:09:00.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337154 2024-02-05T06:09:00.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337154 2024-02-05T06:09:00.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533381 1-481036337154' 2024-02-05T06:09:00.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:09:00.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533381 2024-02-05T06:09:00.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:09:00.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:09:00.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533381 2024-02-05T06:09:00.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:09:00.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533381 2024-02-05T06:09:00.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533381' 2024-02-05T06:09:00.874 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 455266533381 2024-02-05T06:09:00.874 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:09:01.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533380 -lt 455266533381 2024-02-05T06:09:01.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:09:02.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:09:02.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:09:02.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533380 -lt 455266533381 2024-02-05T06:09:02.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:09:03.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:09:03.476 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:09:03.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533381 -lt 455266533381 2024-02-05T06:09:03.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:09:03.783 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-481036337154 2024-02-05T06:09:03.784 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:09:03.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:09:03.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-481036337154 2024-02-05T06:09:03.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:09:03.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337154 2024-02-05T06:09:03.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 481036337154' 2024-02-05T06:09:03.789 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 481036337154 2024-02-05T06:09:03.789 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:09:04.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337154 -lt 481036337154 2024-02-05T06:09:04.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:09:04.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:04.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:04.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:09:04.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:09:04.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:09:04.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:09:04.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:09:04.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:09:04.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:09:04.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:09:04.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:09:04.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:09:04.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:04.874 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:05.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:09:05.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:09:05.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:09:05.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-02-05T06:09:05.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-02-05T06:09:05.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:09:05.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:09:05.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:09:05.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:09:05.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-02-05T06:09:05.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:09:05.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:09:05.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:09:05.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:09:05.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:09:05.236 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:09:05.236 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:09:05.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:09:05.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:09:05.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:09:05.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:09:05.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-02-05T06:09:05.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:09:05.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:09:05.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:09:05.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:09:05.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:09:05.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-02-05T06:09:07.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:09:07.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:09:07.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:09:07.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:09:07.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:09:07.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:09:07.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:09:07.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:09:07.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:09:07.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:09:07.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:09:07.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:09:07.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:09:07.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:09:07.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:09:07.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:09:07.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:09:07.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:09:07.891 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:09:07.891 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:09:07.891 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:09:07.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:09:07.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:09:07.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:09:07.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:09:07.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:09:07.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:09:07.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:09:07.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:09:07.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:09:07.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:09:07.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:09:07.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:09:07.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:09:07.894 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:09:07.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:09:07.894 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:09:07.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:09:07.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:09:07.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:09:07.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:09:07.927 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:07.952+0000 7f319daab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:07.930 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:07.956+0000 7f319daab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:07.932 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:07.956+0000 7f319daab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:08.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:09:08.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:09:08.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:09:08.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:09:08.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:09:08.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:08.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:09:08.205 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:09:08.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:08.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:08.467 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:08.492+0000 7f319daab640 -1 Falling back to public interface 2024-02-05T06:09:08.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:09.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:09.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:09.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:09:09.486 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:09:09.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:09.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:09.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:10.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:10.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:10.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:09:10.804 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:09:10.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:10.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:11.057 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:11.080+0000 7f319daab640 -1 osd.1 113 log_to_monitors true 2024-02-05T06:09:11.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:12.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:12.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:12.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:09:12.126 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:09:12.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:12.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:12.605 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 117 up_thru 117 down_at 114 last_clean_interval [112,113) [v2:127.0.0.1:6810/1522766801,v1:127.0.0.1:6811/1522766801] [v2:127.0.0.1:6812/1522766801,v1:127.0.0.1:6813/1522766801] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:09:12.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:09:12.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:09:12.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:09:12.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:09:12.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:09:12.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:09:12.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:09:12.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:09:12.606 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:09:12.606 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:09:12.607 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:09:12.607 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:09:12.607 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:09:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:09:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:09:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:09:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:09:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:09:12.723 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:09:13.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:09:13.030 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:09:13.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:09:13.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:09:13.031 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:09:13.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533385 2024-02-05T06:09:13.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533385 2024-02-05T06:09:13.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533385' 2024-02-05T06:09:13.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:09:13.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:09:13.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=502511173634 2024-02-05T06:09:13.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 502511173634 2024-02-05T06:09:13.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533385 1-502511173634' 2024-02-05T06:09:13.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:09:13.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533385 2024-02-05T06:09:13.359 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:09:13.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:09:13.361 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533385 2024-02-05T06:09:13.361 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:09:13.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533385 2024-02-05T06:09:13.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533385' 2024-02-05T06:09:13.362 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 455266533385 2024-02-05T06:09:13.363 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:09:13.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533383 -lt 455266533385 2024-02-05T06:09:13.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:09:14.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:09:14.676 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:09:14.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533385 -lt 455266533385 2024-02-05T06:09:14.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:09:14.996 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-502511173634 2024-02-05T06:09:14.997 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:09:14.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:09:14.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-502511173634 2024-02-05T06:09:14.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:09:15.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=502511173634 2024-02-05T06:09:15.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 502511173634' 2024-02-05T06:09:15.001 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 502511173634 2024-02-05T06:09:15.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:09:15.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 502511173634 -lt 502511173634 2024-02-05T06:09:15.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:09:15.338 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:15.338 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:15.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:09:15.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:09:15.725 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:09:15.725 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:09:15.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:09:15.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:09:15.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:09:15.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:09:16.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:09:16.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:09:16.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:16.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:16.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:09:16.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:09:16.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:09:16.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-02-05T06:09:16.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:09:16.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-02-05T06:09:16.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-02-05T06:09:16.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T06:09:16.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:09:16.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-02-05T06:09:16.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:09:16.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:09:16.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:09:16.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:09:16.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-02-05T06:09:16.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:09:16.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:09:16.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:09:16.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:09:16.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:09:16.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:09:16.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:09:16.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:09:16.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:09:16.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:09:16.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:09:16.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-02-05T06:09:16.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:09:16.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:09:16.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:09:16.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:09:16.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:09:16.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-02-05T06:09:19.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:09:19.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:09:19.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:09:19.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:09:19.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:09:19.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:09:19.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:09:19.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:09:19.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:09:19.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:09:19.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:09:19.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:09:19.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:09:19.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:09:19.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:09:19.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:09:19.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:09:19.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:09:19.139 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:09:19.139 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:09:19.139 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:09:19.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:09:19.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:09:19.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:09:19.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:09:19.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:09:19.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:09:19.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:09:19.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:09:19.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:09:19.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:09:19.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:09:19.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:09:19.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:09:19.142 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:09:19.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:09:19.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:09:19.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:09:19.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:09:19.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:09:19.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:09:19.174 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:19.200+0000 7fcd401de640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:19.177 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:19.204+0000 7fcd401de640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:19.178 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:19.204+0000 7fcd401de640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:19.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:09:19.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:09:19.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:09:19.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:09:19.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:09:19.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:19.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:09:19.494 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:09:19.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:19.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:19.715 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:19.740+0000 7fcd401de640 -1 Falling back to public interface 2024-02-05T06:09:19.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:20.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:20.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:20.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:09:20.832 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:09:20.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:20.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:21.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:22.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:22.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:22.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:09:22.149 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:09:22.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:22.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:22.232 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:22.256+0000 7fcd401de640 -1 osd.1 119 log_to_monitors true 2024-02-05T06:09:22.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:23.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:23.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:23.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:09:23.555 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:09:23.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:23.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:23.892 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 123 up_thru 117 down_at 120 last_clean_interval [117,119) [v2:127.0.0.1:6810/1677984174,v1:127.0.0.1:6811/1677984174] [v2:127.0.0.1:6812/1677984174,v1:127.0.0.1:6813/1677984174] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:09:23.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:09:23.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:09:23.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:09:23.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:09:23.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:09:23.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:09:23.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:09:23.893 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:09:23.894 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:09:23.894 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:09:23.894 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:09:23.894 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:09:23.894 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:09:24.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:09:24.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:09:24.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:09:24.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:09:24.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:09:24.005 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:09:24.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:09:24.353 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:09:24.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:09:24.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:09:24.354 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:09:24.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533388 2024-02-05T06:09:24.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533388 2024-02-05T06:09:24.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533388' 2024-02-05T06:09:24.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:09:24.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:09:24.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977410 2024-02-05T06:09:24.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977410 2024-02-05T06:09:24.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533388 1-528280977410' 2024-02-05T06:09:24.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:09:24.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533388 2024-02-05T06:09:24.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:09:24.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:09:24.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533388 2024-02-05T06:09:24.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:09:24.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533388 2024-02-05T06:09:24.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533388' 2024-02-05T06:09:24.691 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 455266533388 2024-02-05T06:09:24.691 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:09:25.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533388 -lt 455266533388 2024-02-05T06:09:25.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:09:25.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-528280977410 2024-02-05T06:09:25.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:09:25.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:09:25.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:09:25.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-528280977410 2024-02-05T06:09:25.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977410 2024-02-05T06:09:25.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 528280977410' 2024-02-05T06:09:25.057 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 528280977410 2024-02-05T06:09:25.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:09:25.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 528280977410 2024-02-05T06:09:25.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:09:26.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:09:26.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:09:26.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977410 -lt 528280977410 2024-02-05T06:09:26.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:09:26.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:26.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:27.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:09:27.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:09:27.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:09:27.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:09:27.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:09:27.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:09:27.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:09:27.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:09:27.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:09:27.504 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:09:27.504 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:27.504 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:27.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:09:27.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:09:27.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:09:27.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-02-05T06:09:27.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-02-05T06:09:28.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T06:09:28.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:09:28.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:09:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:09:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:09:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T06:09:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:09:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:09:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:09:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:09:28.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:09:28.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:09:28.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:09:28.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:09:28.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:09:28.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:09:28.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:09:28.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T06:09:28.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:09:28.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:09:28.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:09:28.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:09:28.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:09:28.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T06:09:30.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:09:30.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:09:30.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:09:30.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:09:30.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:09:30.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:09:30.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:09:30.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:09:30.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:09:30.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:09:30.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:09:30.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:09:30.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:09:30.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:09:30.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:09:30.838 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:09:30.838 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:09:30.838 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:09:30.839 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:09:30.839 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:09:30.839 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:09:30.839 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:09:30.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:09:30.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:09:30.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:09:30.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:09:30.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:09:30.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:09:30.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:09:30.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:09:30.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:09:30.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:09:30.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:09:30.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:09:30.841 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:09:30.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:09:30.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:09:30.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:09:30.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:09:30.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:09:30.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:09:30.876 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:30.900+0000 7f583d025640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:30.879 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:30.904+0000 7f583d025640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:30.880 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:30.904+0000 7f583d025640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:31.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:09:31.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:09:31.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:09:31.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:09:31.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:09:31.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:31.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:09:31.153 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:09:31.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:31.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:31.415 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:31.440+0000 7f583d025640 -1 Falling back to public interface 2024-02-05T06:09:31.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:32.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:32.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:32.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:09:32.461 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:09:32.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:32.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:32.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:33.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:33.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:33.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:09:33.779 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:09:33.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:33.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:33.933 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:33.956+0000 7f583d025640 -1 osd.1 125 log_to_monitors true 2024-02-05T06:09:34.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:35.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:35.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:35.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:09:35.099 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:09:35.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:35.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:35.672 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 129 up_thru 129 down_at 126 last_clean_interval [123,125) [v2:127.0.0.1:6810/1271819987,v1:127.0.0.1:6811/1271819987] [v2:127.0.0.1:6812/1271819987,v1:127.0.0.1:6813/1271819987] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:09:35.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:09:35.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:09:35.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:09:35.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:09:35.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:09:35.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:09:35.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:09:35.673 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:09:35.673 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:09:35.673 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:09:35.673 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:09:35.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:09:35.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:09:35.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:09:35.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:09:35.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:09:35.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:09:35.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:09:35.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:09:36.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:09:36.115 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:09:36.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:09:36.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:09:36.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:09:36.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533391 2024-02-05T06:09:36.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533391 2024-02-05T06:09:36.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533391' 2024-02-05T06:09:36.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:09:36.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:09:36.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=554050781186 2024-02-05T06:09:36.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 554050781186 2024-02-05T06:09:36.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533391 1-554050781186' 2024-02-05T06:09:36.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:09:36.409 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:09:36.409 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533391 2024-02-05T06:09:36.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:09:36.410 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533391 2024-02-05T06:09:36.410 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:09:36.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533391 2024-02-05T06:09:36.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533391' 2024-02-05T06:09:36.412 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 455266533391 2024-02-05T06:09:36.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:09:36.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533390 -lt 455266533391 2024-02-05T06:09:36.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:09:37.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:09:37.708 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:09:38.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533391 -lt 455266533391 2024-02-05T06:09:38.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:09:38.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-554050781186 2024-02-05T06:09:38.027 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:09:38.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:09:38.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-554050781186 2024-02-05T06:09:38.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:09:38.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=554050781186 2024-02-05T06:09:38.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 554050781186' 2024-02-05T06:09:38.032 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 554050781186 2024-02-05T06:09:38.032 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:09:38.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 554050781186 -lt 554050781186 2024-02-05T06:09:38.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:09:38.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:38.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:38.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:09:38.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:09:38.764 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:09:38.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:09:38.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:09:38.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:09:38.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:09:38.765 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:09:39.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:09:39.063 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:09:39.063 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:39.063 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:09:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:09:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:09:39.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-02-05T06:09:39.421 INFO:tasks.workunit.client.0.smithi148.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-02-05T06:09:39.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:09:39.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-02-05T06:09:39.421 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-02-05T06:09:39.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T06:09:39.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:09:39.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:09:39.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-02-05T06:09:39.423 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-02-05T06:09:39.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T06:09:39.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:09:39.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:09:39.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-02-05T06:09:39.424 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-02-05T06:09:39.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T06:09:39.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:09:39.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:09:39.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-02-05T06:09:39.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-02-05T06:09:39.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T06:09:39.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:09:39.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:09:39.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-02-05T06:09:39.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-02-05T06:09:39.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T06:09:39.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:09:39.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-02-05T06:09:39.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-02-05T06:09:39.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:09:39.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:09:39.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:09:39.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:09:39.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-02-05T06:09:39.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:09:39.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:09:39.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:09:39.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:09:39.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:09:39.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:09:39.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:09:39.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:09:39.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:09:39.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:09:39.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:09:39.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-02-05T06:09:39.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:09:39.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:09:39.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:09:39.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:09:39.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:09:39.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-02-05T06:09:41.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:09:41.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:09:41.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:09:41.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:09:41.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:09:41.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:09:41.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:09:41.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:09:41.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:09:41.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:09:41.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:09:41.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:09:41.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:09:41.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:09:41.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:09:41.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:09:41.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:09:41.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:09:41.803 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:09:41.803 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:09:41.803 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:09:41.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:09:41.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:09:41.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:09:41.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:09:41.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:09:41.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:09:41.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:09:41.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:09:41.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:09:41.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:09:41.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:09:41.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:09:41.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:09:41.804 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:09:41.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:09:41.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:09:41.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:09:41.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:09:41.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:09:41.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:09:41.844 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:41.867+0000 7fa742539640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:41.847 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:41.871+0000 7fa742539640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:41.848 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:41.871+0000 7fa742539640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:42.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:09:42.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:09:42.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:09:42.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:09:42.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:09:42.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:42.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:09:42.125 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:09:42.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:42.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:09:42.412 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:42.435+0000 7fa742539640 -1 Falling back to public interface 2024-02-05T06:09:42.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:43.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:43.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:43.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:09:43.438 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:09:43.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:43.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:09:43.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:44.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:44.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:44.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:09:44.747 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:09:44.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:44.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:09:45.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:45.452 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:45.475+0000 7fa742539640 -1 osd.0 131 log_to_monitors true 2024-02-05T06:09:46.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:46.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:46.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:09:46.100 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:09:46.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:46.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:09:46.501 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 135 up_thru 130 down_at 132 last_clean_interval [106,131) [v2:127.0.0.1:6802/1616070518,v1:127.0.0.1:6803/1616070518] [v2:127.0.0.1:6804/1616070518,v1:127.0.0.1:6805/1616070518] exists,up fbea8a79-eb9c-42b5-b2ac-96fd14b4911d 2024-02-05T06:09:46.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:09:46.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:09:46.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:09:46.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:09:46.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:09:46.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:09:46.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:09:46.503 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:09:46.503 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:09:46.503 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:09:46.503 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:09:46.503 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:09:46.503 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:09:46.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:09:46.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:09:46.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:09:46.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:09:46.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:09:46.616 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:09:47.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:09:47.070 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:09:47.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:09:47.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:09:47.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:09:47.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=579820584962 2024-02-05T06:09:47.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 579820584962 2024-02-05T06:09:47.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584962' 2024-02-05T06:09:47.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:09:47.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:09:47.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=554050781189 2024-02-05T06:09:47.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 554050781189 2024-02-05T06:09:47.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584962 1-554050781189' 2024-02-05T06:09:47.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:09:47.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-579820584962 2024-02-05T06:09:47.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:09:47.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:09:47.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-579820584962 2024-02-05T06:09:47.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:09:47.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=579820584962 2024-02-05T06:09:47.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 579820584962' 2024-02-05T06:09:47.473 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 579820584962 2024-02-05T06:09:47.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:09:47.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 579820584962 2024-02-05T06:09:47.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:09:48.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:09:48.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:09:49.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584962 -lt 579820584962 2024-02-05T06:09:49.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:09:49.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-554050781189 2024-02-05T06:09:49.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:09:49.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:09:49.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-554050781189 2024-02-05T06:09:49.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:09:49.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=554050781189 2024-02-05T06:09:49.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 554050781189' 2024-02-05T06:09:49.182 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 554050781189 2024-02-05T06:09:49.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:09:49.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 554050781189 -lt 554050781189 2024-02-05T06:09:49.490 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:09:49.491 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:49.491 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:49.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:09:49.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:09:49.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:09:49.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:09:49.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:09:49.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:09:49.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:09:49.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:09:50.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T06:09:50.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:09:50.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:50.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:50.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 6 2024-02-05T06:09:50.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-02-05T06:09:50.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:09:50.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-02-05T06:09:50.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:09:50.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:09:50.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:09:50.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:09:50.597 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:09:50.597 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:09:50.597 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:09:50.597 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:09:50.597 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:09:50.597 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:09:50.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:09:50.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:09:50.908 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:50.908 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:51.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T06:09:51.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 2 2024-02-05T06:09:51.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:09:51.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T06:09:51.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:09:51.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:09:51.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:09:51.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:09:51.377 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:09:51.377 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:09:51.377 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:09:51.377 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:09:51.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:09:51.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:09:51.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:09:51.665 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:09:51.665 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:51.666 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:52.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T06:09:52.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T06:09:52.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:09:52.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:09:52.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:09:52.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:09:52.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:09:52.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:09:52.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:09:52.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:09:52.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:09:52.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:09:52.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:09:52.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:09:52.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:09:52.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:09:52.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:09:52.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:09:52.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:09:52.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:09:52.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:09:52.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:09:52.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:09:52.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:09:52.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:52.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:53.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:09:53.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:09:53.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:09:53.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-02-05T06:09:53.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:09:53.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:09:53.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:09:53.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:09:53.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-02-05T06:09:53.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:09:53.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:09:53.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:09:53.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:09:53.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:09:53.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:09:53.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:09:53.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:09:53.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:09:53.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:09:53.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:09:53.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-02-05T06:09:53.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:09:53.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:09:53.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:09:53.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:09:53.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:09:53.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-02-05T06:09:55.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:09:55.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:09:55.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:09:55.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:09:55.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:09:55.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:09:55.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:09:55.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:09:55.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:09:55.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:09:55.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:09:55.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:09:55.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:09:55.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:09:55.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:09:55.666 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:09:55.666 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:09:55.666 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:09:55.666 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:09:55.666 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:09:55.666 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:09:55.666 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:09:55.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:09:55.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:09:55.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:09:55.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:09:55.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:09:55.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:09:55.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:09:55.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:09:55.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:09:55.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:09:55.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:09:55.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:09:55.670 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:09:55.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:09:55.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:09:55.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:09:55.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:09:55.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:09:55.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:09:55.700 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:55.723+0000 7f4a33ed0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:55.702 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:55.727+0000 7f4a33ed0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:55.704 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:55.727+0000 7f4a33ed0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:55.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:09:55.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:09:55.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:09:55.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:09:55.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:09:55.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:55.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:09:55.984 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:09:55.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:55.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:56.247 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:56.271+0000 7f4a33ed0640 -1 Falling back to public interface 2024-02-05T06:09:56.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:57.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:57.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:57.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:09:57.276 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:09:57.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:57.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:57.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:58.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:58.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:58.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:09:58.591 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:09:58.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:58.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:09:58.767 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:09:58.791+0000 7f4a33ed0640 -1 osd.1 136 log_to_monitors true 2024-02-05T06:09:58.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:59.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:59.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:59.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:09:59.908 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:09:59.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:59.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:10:00.263 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 140 up_thru 140 down_at 137 last_clean_interval [129,136) [v2:127.0.0.1:6810/3837859053,v1:127.0.0.1:6811/3837859053] [v2:127.0.0.1:6812/3837859053,v1:127.0.0.1:6813/3837859053] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:10:00.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:10:00.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:10:00.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:10:00.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:10:00.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:10:00.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:10:00.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:10:00.265 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:10:00.265 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:10:00.265 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:10:00.265 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:10:00.265 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:10:00.265 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:10:00.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:10:00.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:10:00.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:10:00.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:10:00.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:10:00.374 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:10:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:10:00.832 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:10:00.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:10:00.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:10:00.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:10:00.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=579820584965 2024-02-05T06:10:00.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 579820584965 2024-02-05T06:10:00.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584965' 2024-02-05T06:10:00.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:10:00.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:10:01.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=601295421442 2024-02-05T06:10:01.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 601295421442 2024-02-05T06:10:01.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584965 1-601295421442' 2024-02-05T06:10:01.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:10:01.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-579820584965 2024-02-05T06:10:01.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:10:01.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:10:01.129 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-579820584965 2024-02-05T06:10:01.129 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:10:01.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=579820584965 2024-02-05T06:10:01.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 579820584965' 2024-02-05T06:10:01.130 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 579820584965 2024-02-05T06:10:01.130 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:10:01.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584964 -lt 579820584965 2024-02-05T06:10:01.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:10:02.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:10:02.464 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:10:02.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584966 -lt 579820584965 2024-02-05T06:10:02.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:10:02.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-601295421442 2024-02-05T06:10:02.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:10:02.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:10:02.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-601295421442 2024-02-05T06:10:02.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:10:02.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=601295421442 2024-02-05T06:10:02.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 601295421442' 2024-02-05T06:10:02.781 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 601295421442 2024-02-05T06:10:02.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:10:03.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 601295421442 -lt 601295421442 2024-02-05T06:10:03.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:10:03.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:03.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:03.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:10:03.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:10:03.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:10:03.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:10:03.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:10:03.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:10:03.476 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:10:03.476 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:10:03.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:10:03.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:10:03.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:03.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:04.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:10:04.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:10:04.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:10:04.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-02-05T06:10:04.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:10:04.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-02-05T06:10:04.172 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-02-05T06:10:04.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T06:10:04.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:10:04.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-02-05T06:10:04.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:10:04.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:10:04.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:10:04.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:10:04.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-02-05T06:10:04.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:10:04.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:10:04.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:10:04.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:10:04.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:10:04.175 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:10:04.175 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:10:04.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:10:04.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:10:04.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:10:05.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:10:05.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-02-05T06:10:05.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:10:05.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:10:05.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:10:05.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:10:05.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:10:05.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-02-05T06:10:06.604 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-02-05T06:10:07.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:10:07.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:10:07.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:10:07.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:10:07.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:10:07.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:10:07.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:10:07.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:10:07.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:10:07.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:10:07.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:10:07.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:10:07.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:10:07.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:10:07.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:10:07.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:10:07.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:10:07.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:10:07.144 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:10:07.144 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:10:07.144 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:10:07.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:10:07.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:10:07.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:10:07.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:10:07.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:10:07.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:10:07.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:10:07.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:10:07.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:10:07.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:10:07.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:10:07.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:10:07.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:10:07.145 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:10:07.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:10:07.146 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:10:07.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:10:07.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:10:07.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:10:07.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:10:07.179 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:07.203+0000 7f901e083640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:07.182 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:07.207+0000 7f901e083640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:07.184 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:07.207+0000 7f901e083640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:07.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:10:07.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:10:07.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:10:07.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:10:07.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:10:07.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:07.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:10:07.507 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:10:07.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:07.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:10:07.719 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:07.743+0000 7f901e083640 -1 Falling back to public interface 2024-02-05T06:10:07.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:08.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:08.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:08.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:10:08.863 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:10:08.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:08.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:10:09.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:10.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:10.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:10.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:10:10.172 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:10:10.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:10.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:10:10.297 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:10.323+0000 7f901e083640 -1 osd.1 142 log_to_monitors true 2024-02-05T06:10:10.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:11.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:11.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:11.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:10:11.475 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:10:11.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:11.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:10:11.981 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 146 up_thru 146 down_at 143 last_clean_interval [140,142) [v2:127.0.0.1:6810/3552991628,v1:127.0.0.1:6811/3552991628] [v2:127.0.0.1:6812/3552991628,v1:127.0.0.1:6813/3552991628] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:10:11.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:10:11.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:10:11.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:10:11.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:10:11.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:10:11.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:10:11.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:10:11.982 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:10:11.983 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:10:11.983 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:10:11.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:10:11.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:10:11.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:10:12.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:10:12.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:10:12.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:10:12.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:10:12.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:10:12.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:10:12.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:10:12.484 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:10:12.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:10:12.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:10:12.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:10:12.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=579820584969 2024-02-05T06:10:12.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 579820584969 2024-02-05T06:10:12.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584969' 2024-02-05T06:10:12.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:10:12.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:10:12.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=627065225218 2024-02-05T06:10:12.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 627065225218 2024-02-05T06:10:12.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584969 1-627065225218' 2024-02-05T06:10:12.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:10:12.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-579820584969 2024-02-05T06:10:12.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:10:12.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:10:12.799 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-579820584969 2024-02-05T06:10:12.799 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:10:12.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=579820584969 2024-02-05T06:10:12.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 579820584969' 2024-02-05T06:10:12.800 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 579820584969 2024-02-05T06:10:12.801 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:10:13.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584968 -lt 579820584969 2024-02-05T06:10:13.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:10:14.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:10:14.111 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:10:14.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584968 -lt 579820584969 2024-02-05T06:10:14.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:10:15.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:10:15.424 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:10:15.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584969 -lt 579820584969 2024-02-05T06:10:15.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:10:15.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-627065225218 2024-02-05T06:10:15.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:10:15.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:10:15.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-627065225218 2024-02-05T06:10:15.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:10:15.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=627065225218 2024-02-05T06:10:15.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 627065225218' 2024-02-05T06:10:15.734 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 627065225218 2024-02-05T06:10:15.735 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:10:16.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 627065225218 -lt 627065225218 2024-02-05T06:10:16.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:10:16.089 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:16.089 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:16.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:10:16.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:10:16.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:10:16.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:10:16.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:10:16.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:10:16.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:10:16.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:10:16.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:10:16.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:10:16.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:16.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:17.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:10:17.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:10:17.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:10:17.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:10:17.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-02-05T06:10:17.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-02-05T06:10:17.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T06:10:17.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:10:17.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-02-05T06:10:17.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:10:17.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:10:17.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:10:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:10:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-02-05T06:10:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:10:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:10:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:10:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:10:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:10:17.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:10:17.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:10:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:10:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:10:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:10:17.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:10:17.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-02-05T06:10:17.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:10:17.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:10:17.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:10:17.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:10:17.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:10:17.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-02-05T06:10:19.082 INFO:tasks.workunit.client.0.smithi148.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-02-05T06:10:20.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:10:20.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:10:20.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:10:20.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:10:20.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:10:20.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:10:20.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:10:20.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:10:20.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:10:20.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:10:20.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:10:20.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:10:20.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:10:20.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:10:20.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:10:20.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:10:20.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:10:20.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:10:20.123 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:10:20.123 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:10:20.123 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:10:20.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:10:20.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:10:20.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:10:20.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:10:20.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:10:20.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:10:20.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:10:20.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:10:20.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:10:20.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:10:20.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:10:20.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:10:20.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:10:20.125 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:10:20.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:10:20.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:10:20.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:10:20.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:10:20.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:10:20.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:10:20.163 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:20.187+0000 7fb078c81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:20.166 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:20.191+0000 7fb078c81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:20.168 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:20.191+0000 7fb078c81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:20.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:10:20.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:10:20.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:10:20.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:10:20.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:10:20.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:20.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:10:20.437 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:10:20.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:20.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:20.703 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:20.726+0000 7fb078c81640 -1 Falling back to public interface 2024-02-05T06:10:20.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:21.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:21.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:21.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:10:21.746 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:10:21.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:21.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:22.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:23.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:23.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:23.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:10:23.058 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:10:23.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:23.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:23.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:23.440 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:23.462+0000 7fb078c81640 -1 osd.0 148 log_to_monitors true 2024-02-05T06:10:24.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:24.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:24.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:10:24.377 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:10:24.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:24.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:24.706 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 152 up_thru 152 down_at 149 last_clean_interval [135,148) [v2:127.0.0.1:6802/2226176126,v1:127.0.0.1:6803/2226176126] [v2:127.0.0.1:6804/2226176126,v1:127.0.0.1:6805/2226176126] exists,up fbea8a79-eb9c-42b5-b2ac-96fd14b4911d 2024-02-05T06:10:24.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:10:24.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:10:24.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:10:24.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:10:24.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:10:24.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:10:24.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:10:24.708 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:10:24.708 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:10:24.708 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:10:24.708 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:10:24.708 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:10:24.708 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:10:24.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:10:24.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:10:24.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:10:24.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:10:24.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:10:24.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:10:25.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:10:25.412 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:10:25.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:10:25.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:10:25.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:10:25.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=652835028994 2024-02-05T06:10:25.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 652835028994 2024-02-05T06:10:25.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028994' 2024-02-05T06:10:25.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:10:25.616 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:10:25.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=627065225222 2024-02-05T06:10:25.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 627065225222 2024-02-05T06:10:25.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028994 1-627065225222' 2024-02-05T06:10:25.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:10:25.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-652835028994 2024-02-05T06:10:25.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:10:25.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:10:25.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-652835028994 2024-02-05T06:10:25.762 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:10:25.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=652835028994 2024-02-05T06:10:25.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 652835028994' 2024-02-05T06:10:25.763 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 652835028994 2024-02-05T06:10:25.763 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:10:26.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 652835028994 2024-02-05T06:10:26.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:10:27.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:10:27.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:10:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 652835028994 -lt 652835028994 2024-02-05T06:10:27.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:10:27.456 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-627065225222 2024-02-05T06:10:27.456 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:10:27.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:10:27.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-627065225222 2024-02-05T06:10:27.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:10:27.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=627065225222 2024-02-05T06:10:27.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 627065225222' 2024-02-05T06:10:27.459 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 627065225222 2024-02-05T06:10:27.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:10:27.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 627065225222 -lt 627065225222 2024-02-05T06:10:27.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:10:27.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:27.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:28.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:10:28.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:10:28.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:10:28.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:10:28.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:10:28.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:10:28.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:10:28.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:10:28.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:10:28.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:10:28.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:28.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:28.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T06:10:28.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T06:10:28.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:10:28.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T06:10:28.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:10:28.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:10:28.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:10:28.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:10:28.919 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:10:28.919 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:10:28.919 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:10:28.919 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:10:28.919 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:10:28.919 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:10:29.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:10:29.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:10:29.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:29.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:29.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T06:10:29.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T06:10:29.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:10:29.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:10:29.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:10:29.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:10:29.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:10:29.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:10:29.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:10:29.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:10:29.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:10:29.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:10:29.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:10:29.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:10:30.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:10:30.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:10:30.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:10:30.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:10:30.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:10:30.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:10:30.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:10:30.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:10:30.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:10:30.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:10:30.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:30.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:30.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T06:10:30.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T06:10:30.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:10:30.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:10:30.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:10:30.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:10:30.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:10:30.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:10:30.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:10:31.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:10:31.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:10:31.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T06:10:31.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:10:31.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T06:10:31.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:10:31.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:10:31.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:10:31.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:10:31.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:10:31.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:10:31.604 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:10:31.604 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:10:31.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:10:31.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:10:31.901 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:31.901 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:32.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T06:10:32.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T06:10:32.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:10:32.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:10:32.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:10:32.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:10:32.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:10:32.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:10:32.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:10:32.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:10:32.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:10:32.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-02-05T06:10:32.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:10:32.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-02-05T06:10:33.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:10:33.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:10:33.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:10:33.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:10:33.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:10:33.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:10:33.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:10:33.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:10:33.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:10:33.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:10:33.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:33.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:34.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:10:34.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:10:34.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:10:34.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-02-05T06:10:34.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-02-05T06:10:34.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:10:34.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:10:34.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:10:34.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:10:34.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-02-05T06:10:34.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:10:34.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:10:34.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:10:34.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:10:34.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:10:34.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:10:34.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:10:34.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:10:34.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:10:34.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:10:35.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:10:35.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-02-05T06:10:35.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:10:35.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:10:35.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:10:35.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:10:35.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:10:35.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-02-05T06:10:36.462 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-02-05T06:10:36.462 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-02-05T06:10:36.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:10:36.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:10:36.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:10:36.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:10:36.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:10:37.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:10:37.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:10:37.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:10:37.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:10:37.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:10:37.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:10:37.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:10:37.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:10:37.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:10:37.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:10:37.002 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:10:37.002 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:10:37.002 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:10:37.002 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:10:37.002 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:10:37.002 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:10:37.002 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:10:37.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:10:37.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:10:37.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:10:37.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:10:37.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:10:37.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:10:37.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:10:37.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:10:37.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:10:37.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:10:37.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:10:37.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:10:37.004 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:10:37.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:10:37.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:10:37.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:10:37.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:10:37.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:10:37.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:10:37.038 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:37.062+0000 7fd96caa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:37.041 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:37.066+0000 7fd96caa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:37.042 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:37.066+0000 7fd96caa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:37.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:10:37.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:10:37.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:10:37.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:10:37.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:10:37.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:37.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:10:37.358 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:10:37.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:37.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:10:37.575 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:37.598+0000 7fd96caa5640 -1 Falling back to public interface 2024-02-05T06:10:37.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:38.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:38.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:38.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:10:38.666 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:10:38.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:38.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:10:38.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:39.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:39.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:39.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:10:39.980 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:10:39.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:39.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:10:40.217 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:40.242+0000 7fd96caa5640 -1 osd.1 154 log_to_monitors true 2024-02-05T06:10:40.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:41.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:41.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:41.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:10:41.372 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:10:41.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:41.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:10:41.759 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 158 up_thru 158 down_at 155 last_clean_interval [146,154) [v2:127.0.0.1:6810/1377773542,v1:127.0.0.1:6811/1377773542] [v2:127.0.0.1:6812/1377773542,v1:127.0.0.1:6813/1377773542] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:10:41.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:10:41.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:10:41.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:10:41.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:10:41.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:10:41.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:10:41.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:10:41.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:10:41.761 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:10:41.761 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:10:41.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:10:41.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:10:41.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:10:41.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:10:41.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:10:41.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:10:41.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:10:41.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:10:41.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:10:42.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:10:42.248 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:10:42.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:10:42.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:10:42.249 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:10:42.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=652835028998 2024-02-05T06:10:42.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 652835028998 2024-02-05T06:10:42.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028998' 2024-02-05T06:10:42.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:10:42.394 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:10:42.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=678604832770 2024-02-05T06:10:42.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 678604832770 2024-02-05T06:10:42.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028998 1-678604832770' 2024-02-05T06:10:42.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:10:42.552 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-652835028998 2024-02-05T06:10:42.552 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:10:42.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:10:42.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-652835028998 2024-02-05T06:10:42.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:10:42.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=652835028998 2024-02-05T06:10:42.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 652835028998' 2024-02-05T06:10:42.555 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 652835028998 2024-02-05T06:10:42.556 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:10:42.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 652835028998 -lt 652835028998 2024-02-05T06:10:42.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:10:42.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-678604832770 2024-02-05T06:10:42.868 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:10:42.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:10:42.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:10:42.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-678604832770 2024-02-05T06:10:42.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=678604832770 2024-02-05T06:10:42.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 678604832770' 2024-02-05T06:10:42.872 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 678604832770 2024-02-05T06:10:42.872 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:10:43.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 678604832770 -lt 678604832770 2024-02-05T06:10:43.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:10:43.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:43.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:43.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:10:43.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:10:43.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:10:43.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:10:43.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:10:43.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:10:43.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:10:43.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:10:43.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:10:43.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:10:43.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:43.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:10:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:10:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:10:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:10:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-02-05T06:10:44.229 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-02-05T06:10:44.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T06:10:44.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:10:44.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-02-05T06:10:44.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-02-05T06:10:44.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T06:10:44.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:10:44.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:10:44.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:10:44.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:10:44.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T06:10:44.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:10:44.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:10:44.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:10:44.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:10:44.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:10:44.234 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:10:44.234 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:10:44.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:10:44.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:10:44.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:10:45.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:10:45.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T06:10:45.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:10:45.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:10:45.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:10:45.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:10:45.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:10:45.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T06:10:46.913 INFO:tasks.workunit.client.0.smithi148.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-02-05T06:10:47.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:10:47.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:10:47.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:10:47.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:10:47.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:10:47.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:10:47.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:10:47.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:10:47.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:10:47.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:10:47.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:10:47.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:10:47.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:10:47.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:10:47.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:10:47.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:10:47.854 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:10:47.854 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:10:47.855 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:10:47.855 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:10:47.855 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:10:47.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:10:47.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:10:47.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:10:47.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:10:47.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:10:47.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:10:47.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:10:47.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:10:47.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:10:47.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:10:47.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:10:47.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:10:47.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:10:47.857 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:10:47.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:10:47.857 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:10:47.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:10:47.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:10:47.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:10:47.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:10:47.890 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:47.918+0000 7fe0b1f4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:47.893 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:47.922+0000 7fe0b1f4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:47.895 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:47.922+0000 7fe0b1f4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:48.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:10:48.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:10:48.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:10:48.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:10:48.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:10:48.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:48.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:10:48.179 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:10:48.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:48.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:48.434 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:48.462+0000 7fe0b1f4e640 -1 Falling back to public interface 2024-02-05T06:10:48.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:49.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:49.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:49.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:10:49.481 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:10:49.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:49.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:49.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:50.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:50.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:50.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:10:50.797 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:10:50.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:50.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:51.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:51.249 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:10:51.234+0000 7fe0b1f4e640 -1 osd.0 160 log_to_monitors true 2024-02-05T06:10:52.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:52.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:52.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:10:52.117 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:10:52.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:52.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:52.573 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 164 up_thru 158 down_at 161 last_clean_interval [152,160) [v2:127.0.0.1:6802/438487616,v1:127.0.0.1:6803/438487616] [v2:127.0.0.1:6804/438487616,v1:127.0.0.1:6805/438487616] exists,up fbea8a79-eb9c-42b5-b2ac-96fd14b4911d 2024-02-05T06:10:52.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:10:52.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:10:52.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:10:52.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:10:52.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:10:52.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:10:52.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:10:52.574 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:10:52.575 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:10:52.575 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:10:52.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:10:52.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:10:52.576 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:10:52.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:10:52.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:10:52.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:10:52.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:10:52.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:10:52.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:10:53.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:10:53.064 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:10:53.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:10:53.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:10:53.065 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:10:53.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=704374636546 2024-02-05T06:10:53.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 704374636546 2024-02-05T06:10:53.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636546' 2024-02-05T06:10:53.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:10:53.211 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:10:53.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=678604832773 2024-02-05T06:10:53.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 678604832773 2024-02-05T06:10:53.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636546 1-678604832773' 2024-02-05T06:10:53.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:10:53.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-704374636546 2024-02-05T06:10:53.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:10:53.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:10:53.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-704374636546 2024-02-05T06:10:53.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:10:53.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=704374636546 2024-02-05T06:10:53.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 704374636546' 2024-02-05T06:10:53.469 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 704374636546 2024-02-05T06:10:53.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:10:53.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 704374636546 2024-02-05T06:10:53.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:10:54.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:10:54.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:10:55.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 704374636546 -lt 704374636546 2024-02-05T06:10:55.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:10:55.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-678604832773 2024-02-05T06:10:55.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:10:55.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:10:55.117 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-678604832773 2024-02-05T06:10:55.117 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:10:55.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=678604832773 2024-02-05T06:10:55.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 678604832773' 2024-02-05T06:10:55.118 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 678604832773 2024-02-05T06:10:55.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:10:55.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 678604832773 -lt 678604832773 2024-02-05T06:10:55.451 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:10:55.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:55.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:55.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:10:55.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:10:55.811 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:10:55.811 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:10:55.811 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:10:55.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:10:55.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:10:55.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:10:56.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T06:10:56.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:10:56.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:56.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:56.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 6 2024-02-05T06:10:56.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-02-05T06:10:56.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:10:56.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-02-05T06:10:56.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:10:56.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:10:56.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:10:56.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:10:56.556 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:10:56.556 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:10:56.556 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:10:56.556 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:10:56.557 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:10:56.557 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:10:56.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:10:56.924 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:10:56.924 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:56.924 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:57.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T06:10:57.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 2 2024-02-05T06:10:57.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:10:57.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T06:10:57.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:10:57.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:10:57.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:10:57.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:10:57.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:10:57.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:10:57.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:10:57.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:10:57.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:10:57.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:10:57.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:10:57.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:10:57.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:57.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:58.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T06:10:58.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T06:10:58.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:10:58.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:10:58.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:10:58.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:10:58.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:10:58.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:10:58.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:10:58.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:10:58.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:10:58.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:10:58.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:10:58.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:10:58.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:10:58.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:10:58.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:10:58.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:10:58.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:10:58.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:10:58.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:10:58.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:10:58.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:10:58.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:10:58.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:58.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:59.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:10:59.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:10:59.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:10:59.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T06:10:59.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:10:59.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:10:59.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:10:59.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:10:59.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T06:10:59.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:10:59.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:10:59.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:10:59.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:10:59.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:10:59.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:10:59.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:10:59.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:10:59.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:10:59.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:10:59.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:10:59.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T06:10:59.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:10:59.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:10:59.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:10:59.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:10:59.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:10:59.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T06:11:00.867 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-02-05T06:11:00.867 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-02-05T06:11:01.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:11:01.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:11:01.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:11:01.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:11:01.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:11:01.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:11:01.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:11:01.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:11:01.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:11:01.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:11:01.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:11:01.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:11:01.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:11:01.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:11:01.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:11:01.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:11:01.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:11:01.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:11:01.450 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:11:01.450 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:11:01.450 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:11:01.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:11:01.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:11:01.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:11:01.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:11:01.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:11:01.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:11:01.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:11:01.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:11:01.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:11:01.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:11:01.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:11:01.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:11:01.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:11:01.452 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:11:01.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:11:01.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:11:01.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:11:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:11:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:11:01.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:11:01.483 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:01.510+0000 7f5ff6d8e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:01.486 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:01.514+0000 7f5ff6d8e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:01.488 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:01.518+0000 7f5ff6d8e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:01.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:11:01.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:11:01.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:11:01.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:11:01.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:11:01.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:01.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:11:01.759 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:11:01.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:01.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:02.031 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:02.058+0000 7f5ff6d8e640 -1 Falling back to public interface 2024-02-05T06:11:02.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:03.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:03.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:03.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:11:03.073 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:11:03.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:03.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:03.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:04.386 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:04.414+0000 7f5ff6d8e640 -1 osd.1 165 log_to_monitors true 2024-02-05T06:11:04.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:04.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:04.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:11:04.422 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:11:04.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:04.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:04.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:05.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:05.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:05.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:11:05.825 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:11:05.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:05.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:06.367 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 169 up_thru 169 down_at 166 last_clean_interval [158,165) [v2:127.0.0.1:6810/1539822295,v1:127.0.0.1:6811/1539822295] [v2:127.0.0.1:6812/1539822295,v1:127.0.0.1:6813/1539822295] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:11:06.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:11:06.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:11:06.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:11:06.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:11:06.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:11:06.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:11:06.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:11:06.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:11:06.369 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:11:06.369 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:11:06.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:11:06.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:11:06.369 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:11:06.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:11:06.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:11:06.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:11:06.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:11:06.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:11:06.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:11:06.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:11:06.848 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:11:06.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:11:06.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:11:06.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:11:06.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=704374636550 2024-02-05T06:11:06.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 704374636550 2024-02-05T06:11:06.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636550' 2024-02-05T06:11:06.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:11:06.992 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:11:07.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=725849473026 2024-02-05T06:11:07.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 725849473026 2024-02-05T06:11:07.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636550 1-725849473026' 2024-02-05T06:11:07.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:11:07.129 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-704374636550 2024-02-05T06:11:07.129 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:11:07.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:11:07.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-704374636550 2024-02-05T06:11:07.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:11:07.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=704374636550 2024-02-05T06:11:07.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 704374636550' 2024-02-05T06:11:07.132 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 704374636550 2024-02-05T06:11:07.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:11:07.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 704374636548 -lt 704374636550 2024-02-05T06:11:07.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:11:08.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:11:08.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:11:08.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 704374636550 -lt 704374636550 2024-02-05T06:11:08.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:11:08.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-725849473026 2024-02-05T06:11:08.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:11:08.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:11:08.888 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-725849473026 2024-02-05T06:11:08.888 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:11:08.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=725849473026 2024-02-05T06:11:08.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 725849473026' 2024-02-05T06:11:08.890 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 725849473026 2024-02-05T06:11:08.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:11:09.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 725849473026 -lt 725849473026 2024-02-05T06:11:09.198 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:11:09.199 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:11:09.199 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:11:09.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:11:09.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:11:09.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:11:09.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:11:09.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:11:09.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:11:09.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:11:09.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:11:09.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:11:09.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:11:09.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:11:09.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:11:10.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:11:10.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:11:10.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:11:10.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:11:10.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-02-05T06:11:10.338 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-02-05T06:11:10.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T06:11:10.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:11:10.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-02-05T06:11:10.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-02-05T06:11:10.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T06:11:10.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:11:10.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:11:10.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:11:10.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:11:10.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T06:11:10.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:11:10.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:11:10.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:11:10.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:11:10.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:11:10.342 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:11:10.342 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:11:10.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:11:10.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:11:10.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:11:10.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:11:10.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T06:11:10.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:11:10.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:11:10.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:11:10.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:11:10.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:11:10.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T06:11:12.137 INFO:tasks.workunit.client.0.smithi148.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-02-05T06:11:13.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:11:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:11:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:11:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:11:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:11:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:11:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:11:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:11:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:11:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:11:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:11:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:11:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:11:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:11:13.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:11:13.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:11:13.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:11:13.024 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:11:13.024 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:11:13.024 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:11:13.024 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:11:13.024 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:11:13.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:11:13.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:11:13.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:11:13.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:11:13.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:11:13.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:11:13.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:11:13.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:11:13.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:11:13.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:11:13.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:11:13.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:11:13.026 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:11:13.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:11:13.027 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:11:13.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:11:13.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:11:13.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:11:13.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:11:13.060 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:13.085+0000 7fe897750640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:13.062 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:13.089+0000 7fe897750640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:13.063 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:13.089+0000 7fe897750640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:13.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:11:13.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:11:13.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:11:13.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:11:13.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:11:13.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:13.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:11:13.356 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:11:13.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:13.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:11:13.610 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:13.637+0000 7fe897750640 -1 Falling back to public interface 2024-02-05T06:11:13.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:14.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:14.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:14.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:11:14.660 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:11:14.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:14.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:11:14.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:15.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:15.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:15.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:11:15.965 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:11:15.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:15.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:11:16.254 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:16.281+0000 7fe897750640 -1 osd.0 171 log_to_monitors true 2024-02-05T06:11:16.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:17.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:17.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:17.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:11:17.290 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:11:17.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:17.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:11:17.776 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 175 up_thru 175 down_at 172 last_clean_interval [164,171) [v2:127.0.0.1:6802/383804762,v1:127.0.0.1:6803/383804762] [v2:127.0.0.1:6804/383804762,v1:127.0.0.1:6805/383804762] exists,up fbea8a79-eb9c-42b5-b2ac-96fd14b4911d 2024-02-05T06:11:17.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:11:17.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:11:17.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:11:17.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:11:17.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:11:17.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:11:17.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:11:17.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:11:17.778 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:11:17.778 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:11:17.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:11:17.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:11:17.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:11:17.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:11:17.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:11:17.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:11:17.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:11:17.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:11:17.897 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:11:18.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:11:18.450 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:11:18.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:11:18.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:11:18.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:11:18.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=751619276802 2024-02-05T06:11:18.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 751619276802 2024-02-05T06:11:18.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276802' 2024-02-05T06:11:18.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:11:18.646 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:11:18.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=725849473029 2024-02-05T06:11:18.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 725849473029 2024-02-05T06:11:18.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276802 1-725849473029' 2024-02-05T06:11:18.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:11:18.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-751619276802 2024-02-05T06:11:18.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:11:18.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:11:18.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-751619276802 2024-02-05T06:11:18.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:11:18.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=751619276802 2024-02-05T06:11:18.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 751619276802' 2024-02-05T06:11:18.799 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 751619276802 2024-02-05T06:11:18.799 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:11:19.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 751619276802 2024-02-05T06:11:19.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:11:20.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:11:20.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:11:20.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 751619276802 2024-02-05T06:11:20.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:11:21.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:11:21.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:11:21.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276802 -lt 751619276802 2024-02-05T06:11:21.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:11:21.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-725849473029 2024-02-05T06:11:21.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:11:21.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:11:21.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-725849473029 2024-02-05T06:11:21.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:11:21.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=725849473029 2024-02-05T06:11:21.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 725849473029' 2024-02-05T06:11:21.744 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 725849473029 2024-02-05T06:11:21.744 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:11:22.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 725849473030 -lt 725849473029 2024-02-05T06:11:22.043 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:11:22.044 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:11:22.044 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:11:22.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:11:22.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:11:22.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:11:22.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:11:22.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:11:22.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:11:22.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:11:22.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:11:22.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:11:22.738 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:11:22.738 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:11:22.738 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:11:23.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:11:23.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:11:23.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:11:23.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T06:11:23.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:11:23.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:11:23.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:11:23.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:11:23.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T06:11:23.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:11:23.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:11:23.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:11:23.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:11:23.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:11:23.089 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:11:23.089 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:11:23.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:11:23.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:11:23.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:11:24.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:11:24.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T06:11:24.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:11:24.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:11:24.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:11:24.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:11:24.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:11:24.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T06:11:25.071 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-02-05T06:11:25.071 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-02-05T06:11:25.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:11:25.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:11:25.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:11:25.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:11:25.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:11:25.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:11:25.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:11:25.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:11:25.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:11:25.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:11:25.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:11:25.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:11:25.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:11:25.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:11:25.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:11:25.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:11:25.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:11:25.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:11:25.637 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:11:25.637 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:11:25.637 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:11:25.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:11:25.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:11:25.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:11:25.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:11:25.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:11:25.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:11:25.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:11:25.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:11:25.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:11:25.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:11:25.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:11:25.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:11:25.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:11:25.640 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:11:25.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:11:25.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:11:25.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:11:25.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:11:25.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:11:25.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:11:25.673 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:25.701+0000 7f992b908640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:25.676 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:25.705+0000 7f992b908640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:25.678 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:25.705+0000 7f992b908640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:25.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:11:25.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:11:25.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:11:25.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:11:25.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:11:25.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:25.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:11:25.957 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:11:25.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:25.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:26.210 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:26.237+0000 7f992b908640 -1 Falling back to public interface 2024-02-05T06:11:26.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:27.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:27.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:27.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:11:27.273 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:11:27.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:27.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:27.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:28.516 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:28.541+0000 7f992b908640 -1 osd.1 177 log_to_monitors true 2024-02-05T06:11:28.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:28.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:28.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:11:28.609 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:11:28.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:28.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:29.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:30.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:30.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:30.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:11:30.029 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:11:30.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:30.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:30.413 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 181 up_thru 181 down_at 178 last_clean_interval [169,177) [v2:127.0.0.1:6810/3322247790,v1:127.0.0.1:6811/3322247790] [v2:127.0.0.1:6812/3322247790,v1:127.0.0.1:6813/3322247790] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:11:30.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:11:30.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:11:30.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:11:30.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:11:30.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:11:30.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:11:30.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:11:30.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:11:30.415 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:11:30.415 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:11:30.415 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:11:30.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:11:30.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:11:30.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:11:30.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:11:30.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:11:30.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:11:30.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:11:30.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:11:30.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:11:30.816 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:11:30.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:11:30.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:11:30.817 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:11:30.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=751619276805 2024-02-05T06:11:30.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 751619276805 2024-02-05T06:11:30.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276805' 2024-02-05T06:11:30.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:11:30.958 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:11:31.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=777389080578 2024-02-05T06:11:31.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 777389080578 2024-02-05T06:11:31.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276805 1-777389080578' 2024-02-05T06:11:31.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:11:31.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-751619276805 2024-02-05T06:11:31.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:11:31.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:11:31.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-751619276805 2024-02-05T06:11:31.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:11:31.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=751619276805 2024-02-05T06:11:31.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 751619276805' 2024-02-05T06:11:31.101 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 751619276805 2024-02-05T06:11:31.102 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:11:31.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276804 -lt 751619276805 2024-02-05T06:11:31.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:11:32.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:11:32.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:11:33.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276804 -lt 751619276805 2024-02-05T06:11:33.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:11:34.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:11:34.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:11:34.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276806 -lt 751619276805 2024-02-05T06:11:34.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:11:34.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-777389080578 2024-02-05T06:11:34.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:11:34.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:11:34.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-777389080578 2024-02-05T06:11:34.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:11:34.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=777389080578 2024-02-05T06:11:34.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 777389080578' 2024-02-05T06:11:34.432 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 777389080578 2024-02-05T06:11:34.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:11:34.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 777389080579 -lt 777389080578 2024-02-05T06:11:34.774 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:11:34.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:11:34.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:11:35.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:11:35.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:11:35.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:11:35.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:11:35.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:11:35.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:11:35.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:11:35.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:11:35.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:11:35.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:11:35.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:11:35.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:11:35.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:11:35.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:11:35.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:11:35.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-02-05T06:11:35.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:11:35.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:11:35.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:11:35.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:11:35.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-02-05T06:11:35.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:11:35.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:11:35.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:11:35.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:11:35.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:11:35.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:11:35.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:11:35.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:11:35.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:11:35.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:11:37.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:11:37.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-02-05T06:11:37.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:11:37.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:11:37.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:11:37.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:11:37.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:11:37.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-02-05T06:11:37.769 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-02-05T06:11:37.769 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-02-05T06:11:38.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:11:38.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:11:38.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:11:38.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:11:38.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:11:38.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:11:38.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:11:38.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:11:38.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:11:38.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:11:38.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:11:38.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:11:38.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:11:38.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:11:38.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:11:38.313 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:11:38.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:11:38.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:11:38.315 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:11:38.315 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:11:38.315 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:11:38.315 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:11:38.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:11:38.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:11:38.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:11:38.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:11:38.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:11:38.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:11:38.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:11:38.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:11:38.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:11:38.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:11:38.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:11:38.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:11:38.317 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:11:38.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:11:38.317 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:11:38.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:11:38.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:11:38.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:11:38.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:11:38.349 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:38.377+0000 7fd906125640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:38.351 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:38.377+0000 7fd906125640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:38.353 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:38.381+0000 7fd906125640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:38.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:11:38.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:11:38.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:11:38.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:11:38.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:11:38.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:38.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:11:38.633 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:11:38.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:38.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:38.887 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:38.913+0000 7fd906125640 -1 Falling back to public interface 2024-02-05T06:11:38.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:39.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:39.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:39.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:11:39.946 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:11:39.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:39.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:40.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:41.226 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:41.253+0000 7fd906125640 -1 osd.1 183 log_to_monitors true 2024-02-05T06:11:41.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:41.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:41.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:11:41.252 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:11:41.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:41.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:41.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:42.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:42.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:42.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:11:42.671 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:11:42.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:42.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:43.065 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 187 up_thru 187 down_at 184 last_clean_interval [181,183) [v2:127.0.0.1:6810/3961411938,v1:127.0.0.1:6811/3961411938] [v2:127.0.0.1:6812/3961411938,v1:127.0.0.1:6813/3961411938] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:11:43.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:11:43.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:11:43.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:11:43.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:11:43.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:11:43.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:11:43.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:11:43.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:11:43.067 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:11:43.067 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:11:43.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:11:43.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:11:43.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:11:43.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:11:43.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:11:43.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:11:43.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:11:43.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:11:43.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:11:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:11:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:11:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:11:43.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:11:43.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:11:43.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=751619276809 2024-02-05T06:11:43.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 751619276809 2024-02-05T06:11:43.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276809' 2024-02-05T06:11:43.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:11:43.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:11:43.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=803158884354 2024-02-05T06:11:43.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 803158884354 2024-02-05T06:11:43.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276809 1-803158884354' 2024-02-05T06:11:43.799 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:11:43.799 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-751619276809 2024-02-05T06:11:43.799 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:11:43.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:11:43.801 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-751619276809 2024-02-05T06:11:43.801 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:11:43.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=751619276809 2024-02-05T06:11:43.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 751619276809' 2024-02-05T06:11:43.802 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 751619276809 2024-02-05T06:11:43.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:11:44.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276808 -lt 751619276809 2024-02-05T06:11:44.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:11:45.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:11:45.095 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:11:45.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276809 -lt 751619276809 2024-02-05T06:11:45.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:11:45.402 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-803158884354 2024-02-05T06:11:45.403 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:11:45.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:11:45.405 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-803158884354 2024-02-05T06:11:45.405 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:11:45.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=803158884354 2024-02-05T06:11:45.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 803158884354' 2024-02-05T06:11:45.407 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 803158884354 2024-02-05T06:11:45.407 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:11:45.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 803158884354 -lt 803158884354 2024-02-05T06:11:45.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:11:45.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:11:45.735 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:11:46.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:11:46.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:11:46.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:11:46.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:11:46.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:11:46.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:11:46.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:11:46.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:11:46.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:11:46.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:11:46.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:11:46.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:11:46.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:11:46.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:11:46.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:11:46.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T06:11:46.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-02-05T06:11:46.821 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-02-05T06:11:46.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T06:11:46.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T06:11:46.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-02-05T06:11:46.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-02-05T06:11:46.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-02-05T06:11:46.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-02-05T06:11:46.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:11:47.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-02-05T06:11:47.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-02-05T06:11:47.320 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: {} 2024-02-05T06:11:47.321 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-02-05T06:11:47.337 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: {} 2024-02-05T06:11:47.338 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-02-05T06:11:47.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-02-05T06:11:47.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T06:11:47.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T06:11:47.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-02-05T06:11:47.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T06:11:47.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T06:11:47.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T06:11:47.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-02-05T06:11:47.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T06:11:47.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T06:11:47.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T06:11:47.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-02-05T06:11:47.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T06:11:47.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-02-05T06:11:47.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T06:11:47.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-02-05T06:11:47.354 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-02-05T06:11:47.354 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:11:47.676 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T06:11:47.676 INFO:tasks.workunit.client.0.smithi148.stderr:0' 2024-02-05T06:11:47.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T06:11:47.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T06:11:47.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T06:11:47.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-02-05T06:11:47.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T06:11:47.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T06:11:47.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-02-05T06:11:47.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T06:11:47.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-02-05T06:11:47.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T06:11:47.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-02-05T06:11:47.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T06:11:47.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T06:11:47.679 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T06:11:47.679 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-02-05T06:11:47.680 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:11:47.680 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:11:47.680 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:11:47.680 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:11:47.681 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:11:47.681 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.1.asok 2024-02-05T06:11:47.681 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61355/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-02-05T06:11:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T06:11:47.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T06:11:47.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T06:11:47.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-02-05T06:11:47.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:11:47.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:11:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:11:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:11:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:11:47.793 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.1.asok 2024-02-05T06:11:47.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T06:11:47.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61355/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-02-05T06:11:47.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-02-05T06:11:47.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T06:11:47.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T06:11:47.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-02-05T06:11:47.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T06:11:47.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T06:11:47.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T06:11:47.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-02-05T06:11:47.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T06:11:47.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T06:11:47.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T06:11:47.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-02-05T06:11:47.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T06:11:47.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-02-05T06:11:47.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T06:11:47.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-02-05T06:11:47.903 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-02-05T06:11:47.903 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:11:48.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T06:11:48.244 INFO:tasks.workunit.client.0.smithi148.stderr:0' 2024-02-05T06:11:48.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T06:11:48.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T06:11:48.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T06:11:48.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-02-05T06:11:48.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T06:11:48.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T06:11:48.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-02-05T06:11:48.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T06:11:48.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-02-05T06:11:48.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T06:11:48.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-02-05T06:11:48.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T06:11:48.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T06:11:48.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T06:11:48.247 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-02-05T06:11:48.247 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T06:11:48.247 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T06:11:48.248 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:11:48.248 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:11:48.248 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:11:48.248 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.0.asok 2024-02-05T06:11:48.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61355/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-02-05T06:11:48.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T06:11:48.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T06:11:48.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T06:11:48.360 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-02-05T06:11:48.360 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T06:11:48.360 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T06:11:48.361 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:11:48.361 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:11:48.361 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:11:48.361 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.0.asok 2024-02-05T06:11:48.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T06:11:48.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61355/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-02-05T06:11:48.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-02-05T06:11:48.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T06:11:48.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T06:11:48.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-02-05T06:11:48.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T06:11:48.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T06:11:48.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T06:11:48.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-02-05T06:11:48.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T06:11:48.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T06:11:48.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T06:11:48.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-02-05T06:11:48.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T06:11:48.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-02-05T06:11:48.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T06:11:48.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-02-05T06:11:48.470 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-02-05T06:11:48.470 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:11:48.770 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T06:11:48.770 INFO:tasks.workunit.client.0.smithi148.stderr:0' 2024-02-05T06:11:48.770 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T06:11:48.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T06:11:48.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T06:11:48.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-02-05T06:11:48.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T06:11:48.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T06:11:48.771 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-02-05T06:11:48.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T06:11:48.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-02-05T06:11:48.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T06:11:48.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-02-05T06:11:48.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T06:11:48.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T06:11:48.773 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T06:11:48.773 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-02-05T06:11:48.773 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T06:11:48.773 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T06:11:48.773 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:11:48.773 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:11:48.773 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:11:48.774 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.0.asok 2024-02-05T06:11:48.774 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61355/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-02-05T06:11:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T06:11:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T06:11:48.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-02-05T06:11:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T06:11:48.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T06:11:48.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T06:11:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:11:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:11:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:11:48.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.0.asok 2024-02-05T06:11:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T06:11:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61355/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-02-05T06:11:48.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-02-05T06:11:48.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T06:11:48.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T06:11:48.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-02-05T06:11:48.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T06:11:48.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T06:11:48.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T06:11:48.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-02-05T06:11:48.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T06:11:48.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T06:11:48.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T06:11:48.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-02-05T06:11:48.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T06:11:48.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-02-05T06:11:48.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T06:11:48.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-02-05T06:11:48.996 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-02-05T06:11:48.996 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:11:49.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T06:11:49.328 INFO:tasks.workunit.client.0.smithi148.stderr:0' 2024-02-05T06:11:49.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T06:11:49.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T06:11:49.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T06:11:49.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-02-05T06:11:49.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T06:11:49.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T06:11:49.329 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-02-05T06:11:49.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T06:11:49.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-02-05T06:11:49.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T06:11:49.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-02-05T06:11:49.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T06:11:49.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T06:11:49.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T06:11:49.333 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-02-05T06:11:49.333 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:11:49.333 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:11:49.333 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:11:49.333 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:11:49.333 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:11:49.334 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.1.asok 2024-02-05T06:11:49.334 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61355/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-02-05T06:11:49.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T06:11:49.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T06:11:49.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T06:11:49.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-02-05T06:11:49.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:11:49.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:11:49.450 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:11:49.450 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:11:49.450 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:11:49.451 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.1.asok 2024-02-05T06:11:49.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T06:11:49.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61355/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-02-05T06:11:49.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-02-05T06:11:49.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-02-05T06:11:49.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-02-05T06:11:49.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:11:49.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:11:49.568 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:11:49.568 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:11:49.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-02-05T06:07:23.468440+0000 2024-02-05T06:11:49.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-02-05T06:11:50.129 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 3.0 on osd.1 to scrub 2024-02-05T06:11:50.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-02-05T06:07:23.468440+0000 2024-02-05T06:11:50.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T06:11:50.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:07:23.468440+0000 2024-02-05T06:11:50.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:11:50.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:11:50.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:11:50.147 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:11:50.147 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:11:50.147 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:11:50.147 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:11:50.147 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:11:50.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:07:23.468440+0000 '>' 2024-02-05T06:07:23.468440+0000 2024-02-05T06:11:50.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:11:51.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:11:51.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:11:51.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:11:51.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:11:51.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:11:51.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:11:51.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:11:51.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:07:23.468440+0000 '>' 2024-02-05T06:07:23.468440+0000 2024-02-05T06:11:51.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:11:52.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:11:52.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:11:52.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:11:52.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:11:52.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:11:52.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:11:52.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:11:53.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:11:51.151556+0000 '>' 2024-02-05T06:07:23.468440+0000 2024-02-05T06:11:53.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:11:53.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-02-05T06:11:53.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-02-05T06:11:53.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-02-05T06:11:53.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-02-05T06:11:53.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-02-05T06:11:53.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-02-05T06:11:53.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-02-05T06:11:53.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-02-05T06:11:53.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-02-05T06:11:53.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-02-05T06:11:53.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-02-05T06:11:53.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-02-05T06:11:53.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-02-05T06:11:53.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-02-05T06:11:53.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-02-05T06:11:53.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-02-05T06:11:53.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-02-05T06:11:53.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-02-05T06:11:53.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-02-05T06:11:53.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-02-05T06:11:53.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-02-05T06:11:53.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-02-05T06:11:53.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-02-05T06:11:53.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-02-05T06:11:53.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:11:53.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-02-05T06:11:53.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-02-05T06:11:53.347 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-02-05T06:11:53.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-02-05T06:11:53.377 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-02-05T06:11:53.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-02-05T06:11:53.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-02-05T06:11:53.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-02-05T06:11:53.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=187 2024-02-05T06:11:53.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-02-05T06:11:53.470 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T06:11:53.470 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T06:11:53.470 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T06:11:53.470 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T06:11:53.470 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T06:11:53.470 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T06:11:53.470 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T06:11:53.470 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T06:11:53.470 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T06:11:53.470 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T06:11:53.470 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T06:11:53.470 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T06:11:53.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-02-05T06:11:53.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T06:11:53.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-02-05T06:11:53.516 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T06:11:53.516 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T06:11:53.516 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T06:11:53.516 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T06:11:53.516 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T06:11:53.516 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T06:11:53.516 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T06:11:53.516 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T06:11:53.516 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T06:11:53.516 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T06:11:53.516 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T06:11:53.517 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-02-05T06:11:53.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-02-05T06:11:53.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T06:11:53.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T06:11:53.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T06:11:53.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-02-05T06:11:53.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-02-05T06:11:53.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-02-05T06:11:53.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-02-05T06:11:53.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-02-05T06:11:53.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T06:11:53.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:11:53.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:11:53.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:11:53.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:11:53.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T06:11:53.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:11:53.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:11:53.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:11:53.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:11:53.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:11:53.646 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:11:53.646 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:11:53.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:11:53.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:11:53.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:11:53.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:11:53.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T06:11:53.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:11:53.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:11:53.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:11:53.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:11:53.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:11:53.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T06:11:55.181 INFO:tasks.workunit.client.0.smithi148.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-02-05T06:11:55.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:11:55.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:11:55.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:11:55.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:11:55.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:11:55.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:11:55.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:11:55.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:11:55.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:11:55.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:11:55.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:11:55.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:11:55.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:11:55.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:11:55.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:11:55.722 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:11:55.722 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:11:55.722 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:11:55.722 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:11:55.722 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:11:55.723 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:11:55.725 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:11:55.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:11:55.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:11:55.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:11:55.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:11:55.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:11:55.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:11:55.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:11:55.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:11:55.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:11:55.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:11:55.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:11:55.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:11:55.727 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:11:55.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:11:55.727 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:11:55.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:11:55.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:11:55.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:11:55.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:11:55.756 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:55.784+0000 7f9a09bce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:55.758 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:55.784+0000 7f9a09bce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:55.760 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:55.788+0000 7f9a09bce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:56.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:11:56.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:11:56.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:11:56.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:11:56.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:11:56.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:56.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:11:56.047 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:11:56.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:11:56.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:56.313 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:56.340+0000 7f9a09bce640 -1 Falling back to public interface 2024-02-05T06:11:56.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:57.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:57.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:57.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:11:57.361 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:11:57.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:57.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:11:57.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:58.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:58.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:58.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:11:58.676 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:11:58.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:58.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:11:58.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:59.003 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:11:59.032+0000 7f9a09bce640 -1 osd.0 189 log_to_monitors true 2024-02-05T06:11:59.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:59.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:59.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:11:59.984 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:11:59.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:59.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:12:00.402 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 193 up_thru 193 down_at 190 last_clean_interval [175,189) [v2:127.0.0.1:6802/1308391061,v1:127.0.0.1:6803/1308391061] [v2:127.0.0.1:6804/1308391061,v1:127.0.0.1:6805/1308391061] exists,up fbea8a79-eb9c-42b5-b2ac-96fd14b4911d 2024-02-05T06:12:00.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:12:00.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:12:00.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:12:00.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:12:00.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:12:00.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:12:00.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:12:00.403 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:12:00.404 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:12:00.404 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:12:00.404 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:12:00.404 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:12:00.404 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:12:00.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:12:00.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:12:00.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:12:00.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:12:00.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:12:00.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:12:00.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:12:00.887 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:12:00.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:12:00.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:12:00.887 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:12:01.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=828928688130 2024-02-05T06:12:01.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 828928688130 2024-02-05T06:12:01.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-828928688130' 2024-02-05T06:12:01.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:12:01.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:12:01.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=803158884358 2024-02-05T06:12:01.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 803158884358 2024-02-05T06:12:01.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-828928688130 1-803158884358' 2024-02-05T06:12:01.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:12:01.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-828928688130 2024-02-05T06:12:01.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:12:01.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:12:01.199 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-828928688130 2024-02-05T06:12:01.200 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:12:01.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=828928688130 2024-02-05T06:12:01.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 828928688130' 2024-02-05T06:12:01.201 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 828928688130 2024-02-05T06:12:01.201 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:12:01.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 828928688130 2024-02-05T06:12:01.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:12:02.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:12:02.503 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:12:02.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 828928688130 -lt 828928688130 2024-02-05T06:12:02.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:12:02.869 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-803158884358 2024-02-05T06:12:02.869 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:12:02.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:12:02.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-803158884358 2024-02-05T06:12:02.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:12:02.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=803158884358 2024-02-05T06:12:02.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 803158884358' 2024-02-05T06:12:02.873 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 803158884358 2024-02-05T06:12:02.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:12:03.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 803158884359 -lt 803158884358 2024-02-05T06:12:03.199 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:12:03.200 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:03.200 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:03.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:12:03.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:12:03.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:12:03.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:12:03.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:12:03.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:12:03.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:12:03.629 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:12:03.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:12:03.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:12:03.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:03.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:04.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:12:04.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:12:04.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:12:04.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-02-05T06:12:04.287 INFO:tasks.workunit.client.0.smithi148.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-02-05T06:12:04.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-02-05T06:12:04.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-02-05T06:12:04.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:12:04.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:12:04.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:12:04.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:12:04.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-02-05T06:12:04.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:12:04.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:12:04.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:12:04.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:12:04.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:12:04.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:12:04.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:12:04.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:12:04.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:12:04.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:12:04.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:12:04.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-02-05T06:12:04.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:12:04.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:12:04.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:12:04.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:12:04.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:12:04.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-02-05T06:12:05.744 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-02-05T06:12:05.821 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-02-05T06:12:06.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:12:06.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:12:06.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:12:06.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:12:06.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:12:06.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:12:06.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:12:06.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:12:06.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:12:06.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:12:06.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:12:06.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:12:06.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:12:06.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:12:06.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:12:06.297 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:12:06.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:12:06.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:12:06.298 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:12:06.298 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:06.298 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:06.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:12:06.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:12:06.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:12:06.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:12:06.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:12:06.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:12:06.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:12:06.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:12:06.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:12:06.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:12:06.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:12:06.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:12:06.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:12:06.299 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:12:06.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:12:06.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:12:06.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:12:06.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:12:06.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:12:06.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:12:06.333 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:06.360+0000 7f1e3b450640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:06.335 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:06.364+0000 7f1e3b450640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:06.337 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:06.364+0000 7f1e3b450640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:06.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:12:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:12:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:12:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:12:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:12:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:12:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:12:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:06.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:12:06.886 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:06.912+0000 7f1e3b450640 -1 Falling back to public interface 2024-02-05T06:12:06.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:12:07.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:12:07.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:07.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:12:07.992 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:12:07.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:07.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:12:08.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:12:09.197 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:09.224+0000 7f1e3b450640 -1 osd.1 195 log_to_monitors true 2024-02-05T06:12:09.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:12:09.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:09.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:12:09.345 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:12:09.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:09.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:12:09.695 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 199 up_thru 194 down_at 196 last_clean_interval [187,195) [v2:127.0.0.1:6810/609436807,v1:127.0.0.1:6811/609436807] [v2:127.0.0.1:6812/609436807,v1:127.0.0.1:6813/609436807] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:12:09.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:12:09.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:12:09.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:12:09.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:12:09.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:12:09.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:12:09.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:12:09.696 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:12:09.697 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:12:09.697 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:12:09.697 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:12:09.697 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:12:09.697 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:12:09.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:12:09.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:12:09.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:12:09.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:12:09.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:12:09.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:12:10.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:12:10.185 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:12:10.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:12:10.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:12:10.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:12:10.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=828928688133 2024-02-05T06:12:10.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 828928688133 2024-02-05T06:12:10.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-828928688133' 2024-02-05T06:12:10.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:12:10.398 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:12:10.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=854698491906 2024-02-05T06:12:10.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 854698491906 2024-02-05T06:12:10.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-828928688133 1-854698491906' 2024-02-05T06:12:10.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:12:10.591 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-828928688133 2024-02-05T06:12:10.592 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:12:10.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:12:10.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-828928688133 2024-02-05T06:12:10.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:12:10.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=828928688133 2024-02-05T06:12:10.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 828928688133' 2024-02-05T06:12:10.595 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 828928688133 2024-02-05T06:12:10.596 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:12:11.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 828928688133 -lt 828928688133 2024-02-05T06:12:11.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:12:11.003 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-854698491906 2024-02-05T06:12:11.003 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:12:11.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:12:11.005 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-854698491906 2024-02-05T06:12:11.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:12:11.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=854698491906 2024-02-05T06:12:11.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 854698491906' 2024-02-05T06:12:11.007 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 854698491906 2024-02-05T06:12:11.007 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:12:11.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 854698491906 -lt 854698491906 2024-02-05T06:12:11.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:12:11.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:11.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:11.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:12:11.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:12:11.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:12:11.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:12:11.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:12:11.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:12:11.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:12:11.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:12:12.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:12:12.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:12:12.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:12.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:12.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T06:12:12.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-02-05T06:12:12.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:12:12.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T06:12:12.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:12:12.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:12:12.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:12:12.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:12:12.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:12:12.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:12:12.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:12:12.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:12:12.510 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:12:12.510 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:12:12.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:12:12.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:12:12.828 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:12.828 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:13.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T06:12:13.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T06:12:13.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:12:13.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:12:13.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:12:13.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:12:13.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:12:13.200 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:12:13.200 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:12:13.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:12:13.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:12:13.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:12:13.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:12:13.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:12:13.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:12:13.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:12:13.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:12:13.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:12:13.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:12:13.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:12:13.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:12:13.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:12:14.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:12:14.093 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:12:14.093 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:14.094 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:14.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T06:12:14.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T06:12:14.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:12:14.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:12:14.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:12:14.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:12:14.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:12:14.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:12:14.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:12:14.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:12:14.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:12:14.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T06:12:14.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:12:14.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T06:12:15.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:12:15.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:12:15.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:12:15.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:12:15.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:12:15.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:12:15.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:12:15.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:12:15.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:12:15.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:12:15.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:15.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:15.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:12:15.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:12:15.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:12:15.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-02-05T06:12:15.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T06:12:15.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:12:15.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:12:15.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:12:15.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:12:15.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T06:12:15.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:12:15.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:12:15.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:12:15.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:12:15.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:12:15.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:12:15.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:12:15.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:12:15.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:12:15.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:12:16.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:12:16.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T06:12:16.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:12:16.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:12:16.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:12:16.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:12:16.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:12:16.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T06:12:17.734 INFO:tasks.workunit.client.0.smithi148.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-02-05T06:12:18.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:12:18.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:12:18.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:12:18.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:12:18.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:12:18.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:12:18.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:12:18.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:12:18.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:12:18.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:12:18.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:12:18.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:12:18.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:12:18.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:12:18.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:12:18.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:12:18.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:12:18.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:12:18.272 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:12:18.272 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:18.272 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:18.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:12:18.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:12:18.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:12:18.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:12:18.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:12:18.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:12:18.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:12:18.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:12:18.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:12:18.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:12:18.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:12:18.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:12:18.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:12:18.274 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:12:18.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:12:18.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:12:18.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:12:18.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:12:18.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:12:18.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:12:18.307 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:18.332+0000 7fd920e63640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:18.310 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:18.336+0000 7fd920e63640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:18.311 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:18.340+0000 7fd920e63640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:18.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:12:18.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:12:18.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:12:18.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:12:18.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:12:18.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:18.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:12:18.586 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:12:18.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:18.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:12:18.854 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:18.880+0000 7fd920e63640 -1 Falling back to public interface 2024-02-05T06:12:18.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:12:19.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:12:19.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:19.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:12:19.982 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:12:19.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:19.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:12:20.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:12:21.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:12:21.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:21.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:12:21.285 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:12:21.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:21.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:12:21.492 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:21.520+0000 7fd920e63640 -1 osd.0 201 log_to_monitors true 2024-02-05T06:12:21.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:12:22.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:12:22.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:22.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:12:22.761 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:12:22.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:22.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:12:23.188 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 205 up_thru 205 down_at 202 last_clean_interval [193,201) [v2:127.0.0.1:6802/4217689296,v1:127.0.0.1:6803/4217689296] [v2:127.0.0.1:6804/4217689296,v1:127.0.0.1:6805/4217689296] exists,up fbea8a79-eb9c-42b5-b2ac-96fd14b4911d 2024-02-05T06:12:23.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:12:23.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:12:23.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:12:23.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:12:23.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:12:23.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:12:23.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:12:23.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:12:23.189 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:12:23.189 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:12:23.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:12:23.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:12:23.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:12:23.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:12:23.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:12:23.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:12:23.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:12:23.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:12:23.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:12:23.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:12:23.669 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:12:23.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:12:23.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:12:23.669 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:12:23.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=880468295682 2024-02-05T06:12:23.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 880468295682 2024-02-05T06:12:23.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295682' 2024-02-05T06:12:23.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:12:23.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:12:23.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=854698491909 2024-02-05T06:12:23.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 854698491909 2024-02-05T06:12:23.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295682 1-854698491909' 2024-02-05T06:12:23.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:12:23.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-880468295682 2024-02-05T06:12:23.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:12:23.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:12:23.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-880468295682 2024-02-05T06:12:23.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:12:23.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=880468295682 2024-02-05T06:12:23.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 880468295682' 2024-02-05T06:12:23.991 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 880468295682 2024-02-05T06:12:23.992 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:12:24.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 880468295682 2024-02-05T06:12:24.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:12:25.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:12:25.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:12:25.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 880468295682 -lt 880468295682 2024-02-05T06:12:25.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:12:25.617 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-854698491909 2024-02-05T06:12:25.617 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:12:25.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:12:25.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-854698491909 2024-02-05T06:12:25.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:12:25.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=854698491909 2024-02-05T06:12:25.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 854698491909' 2024-02-05T06:12:25.621 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 854698491909 2024-02-05T06:12:25.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:12:25.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 854698491910 -lt 854698491909 2024-02-05T06:12:25.949 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:12:25.950 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:25.950 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:26.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:12:26.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:12:26.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:12:26.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:12:26.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:12:26.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:12:26.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:12:26.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:12:26.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:12:26.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:12:26.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:26.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:27.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:12:27.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:12:27.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:12:27.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T06:12:27.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:12:27.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:12:27.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:12:27.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:12:27.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T06:12:27.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:12:27.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:12:27.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:12:27.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:12:27.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:12:27.037 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:12:27.037 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:12:27.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:12:27.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:12:27.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:12:27.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:12:27.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T06:12:27.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:12:27.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:12:27.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:12:27.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:12:27.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:12:27.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T06:12:28.527 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-02-05T06:12:28.527 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-02-05T06:12:29.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:12:29.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:12:29.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:12:29.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:12:29.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:12:29.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:12:29.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:12:29.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:12:29.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:12:29.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:12:29.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:12:29.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:12:29.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:12:29.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:12:29.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:12:29.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:12:29.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:12:29.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:12:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:12:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:29.071 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:12:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:12:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:12:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:12:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:12:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:12:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:12:29.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:12:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:12:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:12:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:12:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:12:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:12:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:12:29.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:12:29.073 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:12:29.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:12:29.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:12:29.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:12:29.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:12:29.105 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:29.132+0000 7f240d969640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:29.108 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:29.136+0000 7f240d969640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:29.109 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:29.136+0000 7f240d969640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:29.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:12:29.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:12:29.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:12:29.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:12:29.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:12:29.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:29.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:12:29.399 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:12:29.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:29.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:12:29.642 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:29.668+0000 7f240d969640 -1 Falling back to public interface 2024-02-05T06:12:29.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:12:30.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:12:30.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:30.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:12:30.731 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:12:30.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:30.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:12:31.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:12:31.979 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:32.008+0000 7f240d969640 -1 osd.1 207 log_to_monitors true 2024-02-05T06:12:32.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:12:32.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:32.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:12:32.043 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:12:32.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:32.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:12:32.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:12:33.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:12:33.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:33.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:12:33.381 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:12:33.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:33.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:12:33.910 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 211 up_thru 211 down_at 208 last_clean_interval [199,207) [v2:127.0.0.1:6810/571838857,v1:127.0.0.1:6811/571838857] [v2:127.0.0.1:6812/571838857,v1:127.0.0.1:6813/571838857] exists,up edd95e57-be98-4562-8431-9f2ffce6a753 2024-02-05T06:12:33.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:12:33.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:12:33.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:12:33.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:12:33.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:12:33.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:12:33.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:12:33.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:12:33.911 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:12:33.911 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:12:33.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:12:33.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:12:33.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:12:34.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:12:34.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:12:34.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:12:34.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:12:34.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:12:34.036 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:12:34.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:12:34.351 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:12:34.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:12:34.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:12:34.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:12:34.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=880468295685 2024-02-05T06:12:34.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 880468295685 2024-02-05T06:12:34.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295685' 2024-02-05T06:12:34.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:12:34.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:12:34.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=906238099458 2024-02-05T06:12:34.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 906238099458 2024-02-05T06:12:34.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295685 1-906238099458' 2024-02-05T06:12:34.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:12:34.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-880468295685 2024-02-05T06:12:34.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:12:34.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:12:34.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-880468295685 2024-02-05T06:12:34.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:12:34.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=880468295685 2024-02-05T06:12:34.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 880468295685' 2024-02-05T06:12:34.673 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 880468295685 2024-02-05T06:12:34.673 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:12:34.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 880468295685 -lt 880468295685 2024-02-05T06:12:34.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:12:34.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-906238099458 2024-02-05T06:12:34.978 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:12:34.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:12:34.981 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-906238099458 2024-02-05T06:12:34.981 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:12:34.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=906238099458 2024-02-05T06:12:34.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 906238099458' 2024-02-05T06:12:34.982 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 906238099458 2024-02-05T06:12:34.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:12:35.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 906238099458 2024-02-05T06:12:35.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:12:36.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:12:36.342 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:12:36.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 906238099458 2024-02-05T06:12:36.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:12:37.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:12:37.653 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:12:37.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 906238099458 -lt 906238099458 2024-02-05T06:12:37.955 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:12:37.955 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:37.955 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:38.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:12:38.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:12:38.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:12:38.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:12:38.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:12:38.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:12:38.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:12:38.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:12:38.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:12:38.626 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:12:38.626 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:38.627 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:38.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:12:38.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:12:38.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:12:38.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-02-05T06:12:38.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-02-05T06:12:38.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T06:12:38.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T06:12:38.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-02-05T06:12:38.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T06:12:38.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T06:12:38.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T06:12:38.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-02-05T06:12:38.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T06:12:38.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T06:12:38.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T06:12:38.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-02-05T06:12:38.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T06:12:38.998 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-02-05T06:12:38.998 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T06:12:38.998 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-02-05T06:12:38.998 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-02-05T06:12:38.998 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:12:39.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T06:12:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:0' 2024-02-05T06:12:39.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T06:12:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T06:12:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T06:12:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-02-05T06:12:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T06:12:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T06:12:39.329 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-02-05T06:12:39.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T06:12:39.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-02-05T06:12:39.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T06:12:39.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-02-05T06:12:39.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T06:12:39.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T06:12:39.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T06:12:39.331 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-02-05T06:12:39.331 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:12:39.331 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:12:39.331 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:12:39.331 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:39.332 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:39.332 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.1.asok 2024-02-05T06:12:39.332 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61355/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-02-05T06:12:39.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T06:12:39.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T06:12:39.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T06:12:39.448 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-02-05T06:12:39.448 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:12:39.448 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:12:39.449 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:12:39.449 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:39.449 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:39.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.1.asok 2024-02-05T06:12:39.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T06:12:39.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61355/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-02-05T06:12:39.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-02-05T06:12:39.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T06:12:39.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T06:12:39.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-02-05T06:12:39.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T06:12:39.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T06:12:39.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T06:12:39.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-02-05T06:12:39.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T06:12:39.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T06:12:39.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T06:12:39.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-02-05T06:12:39.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T06:12:39.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-02-05T06:12:39.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T06:12:39.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-02-05T06:12:39.559 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-02-05T06:12:39.559 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:12:39.874 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T06:12:39.874 INFO:tasks.workunit.client.0.smithi148.stderr:0' 2024-02-05T06:12:39.874 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T06:12:39.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T06:12:39.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T06:12:39.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-02-05T06:12:39.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T06:12:39.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T06:12:39.875 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-02-05T06:12:39.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T06:12:39.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-02-05T06:12:39.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T06:12:39.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-02-05T06:12:39.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T06:12:39.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T06:12:39.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T06:12:39.877 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-02-05T06:12:39.877 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T06:12:39.877 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T06:12:39.878 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:12:39.878 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:39.878 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:39.878 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.0.asok 2024-02-05T06:12:39.879 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61355/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-02-05T06:12:39.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T06:12:39.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T06:12:39.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-02-05T06:12:39.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T06:12:39.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T06:12:39.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T06:12:39.995 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:12:39.995 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:39.995 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:39.996 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.0.asok 2024-02-05T06:12:39.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T06:12:39.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61355/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-02-05T06:12:40.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-02-05T06:12:40.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T06:12:40.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T06:12:40.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-02-05T06:12:40.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T06:12:40.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T06:12:40.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T06:12:40.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-02-05T06:12:40.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T06:12:40.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T06:12:40.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T06:12:40.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-02-05T06:12:40.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T06:12:40.105 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-02-05T06:12:40.105 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T06:12:40.105 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-02-05T06:12:40.106 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-02-05T06:12:40.106 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:12:40.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T06:12:40.397 INFO:tasks.workunit.client.0.smithi148.stderr:0' 2024-02-05T06:12:40.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T06:12:40.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T06:12:40.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T06:12:40.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-02-05T06:12:40.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T06:12:40.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T06:12:40.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-02-05T06:12:40.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T06:12:40.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-02-05T06:12:40.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T06:12:40.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-02-05T06:12:40.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T06:12:40.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T06:12:40.399 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T06:12:40.399 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-02-05T06:12:40.399 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T06:12:40.400 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T06:12:40.400 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:12:40.400 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:40.400 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:40.401 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.0.asok 2024-02-05T06:12:40.401 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61355/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-02-05T06:12:40.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T06:12:40.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T06:12:40.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T06:12:40.519 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-02-05T06:12:40.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T06:12:40.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T06:12:40.520 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:12:40.521 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:40.521 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:40.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.0.asok 2024-02-05T06:12:40.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T06:12:40.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61355/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-02-05T06:12:40.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-02-05T06:12:40.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T06:12:40.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T06:12:40.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-02-05T06:12:40.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T06:12:40.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T06:12:40.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T06:12:40.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-02-05T06:12:40.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T06:12:40.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T06:12:40.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T06:12:40.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-02-05T06:12:40.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T06:12:40.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-02-05T06:12:40.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T06:12:40.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-02-05T06:12:40.640 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-02-05T06:12:40.640 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:12:40.940 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T06:12:40.940 INFO:tasks.workunit.client.0.smithi148.stderr:0' 2024-02-05T06:12:40.940 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T06:12:40.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T06:12:40.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T06:12:40.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-02-05T06:12:40.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T06:12:40.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T06:12:40.941 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-02-05T06:12:40.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T06:12:40.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-02-05T06:12:40.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T06:12:40.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-02-05T06:12:40.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T06:12:40.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T06:12:40.943 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T06:12:40.943 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-02-05T06:12:40.943 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:12:40.943 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:12:40.944 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:12:40.944 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:40.944 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:40.944 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.1.asok 2024-02-05T06:12:40.945 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61355/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-02-05T06:12:41.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T06:12:41.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T06:12:41.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T06:12:41.062 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-02-05T06:12:41.062 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:12:41.062 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:12:41.063 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:12:41.063 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:41.063 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:41.063 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-osd.1.asok 2024-02-05T06:12:41.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T06:12:41.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61355/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-02-05T06:12:41.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-02-05T06:12:41.302 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: {} 2024-02-05T06:12:41.302 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0: osd_max_object_size = '' 2024-02-05T06:12:41.319 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: {} 2024-02-05T06:12:41.320 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1: osd_max_object_size = '' 2024-02-05T06:12:41.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-02-05T06:12:41.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-02-05T06:12:41.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:12:41.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:12:41.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:12:41.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:41.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:12:41.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-02-05T06:07:23.468440+0000 2024-02-05T06:12:41.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-02-05T06:12:41.897 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-02-05T06:12:41.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-02-05T06:07:23.468440+0000 last_deep_scrub_stamp 2024-02-05T06:12:41.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T06:12:41.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:07:23.468440+0000 2024-02-05T06:12:41.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-02-05T06:12:41.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:12:41.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:12:41.914 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:12:41.914 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:12:41.914 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:12:41.914 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:41.914 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:12:42.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:07:23.468440+0000 '>' 2024-02-05T06:07:23.468440+0000 2024-02-05T06:12:42.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:12:43.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:12:43.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:12:43.194 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T06:12:43.194 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:12:43.194 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:12:43.195 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:43.195 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T06:12:43.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:12:42.115516+0000 '>' 2024-02-05T06:07:23.468440+0000 2024-02-05T06:12:43.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:12:43.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-02-05T06:12:43.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-02-05T06:12:43.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-02-05T06:12:43.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-02-05T06:12:43.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-02-05T06:12:43.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-02-05T06:12:43.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-02-05T06:12:43.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-02-05T06:12:43.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-02-05T06:12:43.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-02-05T06:12:43.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-02-05T06:12:43.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-02-05T06:12:43.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-02-05T06:12:43.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-02-05T06:12:43.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-02-05T06:12:43.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-02-05T06:12:43.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-02-05T06:12:43.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-02-05T06:12:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-02-05T06:12:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-02-05T06:12:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-02-05T06:12:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-02-05T06:12:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-02-05T06:12:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-02-05T06:12:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-02-05T06:12:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-02-05T06:12:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-02-05T06:12:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-02-05T06:12:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-02-05T06:12:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-02-05T06:12:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-02-05T06:12:43.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-02-05T06:12:43.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-02-05T06:12:43.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-02-05T06:12:43.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-02-05T06:12:43.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-02-05T06:12:43.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-02-05T06:12:43.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-02-05T06:12:43.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-02-05T06:12:43.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-02-05T06:12:43.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T06:12:43.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-02-05T06:12:43.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-02-05T06:12:44.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-02-05T06:12:44.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-02-05T06:12:44.031 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-02-05T06:12:44.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-02-05T06:12:44.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-02-05T06:12:44.089 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-02-05T06:12:44.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=211 2024-02-05T06:12:44.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-02-05T06:12:44.119 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T06:12:44.119 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T06:12:44.119 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T06:12:44.119 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T06:12:44.119 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T06:12:44.119 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T06:12:44.119 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T06:12:44.119 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T06:12:44.120 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T06:12:44.120 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T06:12:44.120 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T06:12:44.120 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T06:12:44.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-02-05T06:12:44.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T06:12:44.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-02-05T06:12:44.174 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T06:12:44.174 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T06:12:44.174 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T06:12:44.174 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T06:12:44.174 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T06:12:44.174 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T06:12:44.174 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T06:12:44.174 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T06:12:44.175 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T06:12:44.175 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T06:12:44.175 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T06:12:44.175 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-02-05T06:12:44.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-02-05T06:12:44.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T06:12:44.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T06:12:44.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T06:12:44.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-02-05T06:12:44.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-02-05T06:12:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-02-05T06:12:44.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=3.0 2024-02-05T06:12:44.229 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 3.0 2024-02-05T06:12:44.229 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:12:44.229 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:12:44.230 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:44.230 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:12:44.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T06:12:42.115516+0000 2024-02-05T06:12:44.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 3.0 2024-02-05T06:12:44.871 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 3.0 on osd.1 to repair 2024-02-05T06:12:44.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 3.0 2024-02-05T06:12:42.115516+0000 2024-02-05T06:12:44.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T06:12:44.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:12:42.115516+0000 2024-02-05T06:12:44.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:12:44.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:12:44.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:12:44.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:12:44.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:12:44.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:12:44.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:44.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:12:45.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:12:42.115516+0000 '>' 2024-02-05T06:12:42.115516+0000 2024-02-05T06:12:45.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:12:46.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:12:46.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:12:46.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:12:46.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:12:46.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:12:46.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:46.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:12:46.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:12:42.115516+0000 '>' 2024-02-05T06:12:42.115516+0000 2024-02-05T06:12:46.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:12:47.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:12:47.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:12:47.527 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:12:47.527 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:12:47.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:12:47.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:47.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:12:47.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:12:42.115516+0000 '>' 2024-02-05T06:12:42.115516+0000 2024-02-05T06:12:47.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:12:48.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:12:48.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:12:48.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:12:48.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:12:48.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:12:48.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:48.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:12:49.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:12:45.931118+0000 '>' 2024-02-05T06:12:42.115516+0000 2024-02-05T06:12:49.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:12:49.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-02-05T06:12:49.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:12:49.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:12:49.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:12:49.163 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:12:49.163 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:12:49.163 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:12:49.164 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:12:49.164 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:12:49.164 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:12:49.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:12:49.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:12:49.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:12:49.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:12:49.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:12:49.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:12:49.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:12:49.582 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:12:49.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:12:49.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:12:49.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:12:49.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=880468295689 2024-02-05T06:12:49.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 880468295689 2024-02-05T06:12:49.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295689' 2024-02-05T06:12:49.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:12:49.744 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:12:49.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=906238099462 2024-02-05T06:12:49.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 906238099462 2024-02-05T06:12:49.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295689 1-906238099462' 2024-02-05T06:12:49.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:12:49.883 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-880468295689 2024-02-05T06:12:49.883 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:12:49.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:12:49.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-880468295689 2024-02-05T06:12:49.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:12:49.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=880468295689 2024-02-05T06:12:49.887 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 880468295689 2024-02-05T06:12:49.887 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 880468295689' 2024-02-05T06:12:49.887 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:12:50.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 880468295688 -lt 880468295689 2024-02-05T06:12:50.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:12:51.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:12:51.198 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:12:51.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 880468295689 -lt 880468295689 2024-02-05T06:12:51.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:12:51.508 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-906238099462 2024-02-05T06:12:51.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:12:51.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:12:51.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-906238099462 2024-02-05T06:12:51.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:12:51.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=906238099462 2024-02-05T06:12:51.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 906238099462' 2024-02-05T06:12:51.513 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 906238099462 2024-02-05T06:12:51.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:12:51.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 906238099462 -lt 906238099462 2024-02-05T06:12:51.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:12:51.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:51.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:52.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:12:52.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:12:52.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:12:52.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:12:52.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:12:52.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:12:52.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:12:52.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:12:52.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:12:52.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:12:52.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:52.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:52.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:12:52.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:12:52.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:12:52.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-02-05T06:12:52.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-02-05T06:12:52.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-02-05T06:12:52.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-02-05T06:12:52.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-02-05T06:12:52.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-02-05T06:12:52.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-02-05T06:12:52.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-02-05T06:12:53.698 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'csr_pool' removed 2024-02-05T06:12:53.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:12:53.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:12:53.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:12:53.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:12:53.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:12:53.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:12:53.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:12:53.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:12:53.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:12:54.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:12:54.035 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:12:54.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:12:54.037 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:12:54.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:12:54.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:12:54.039 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:12:54.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:12:54.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:12:54.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:12:54.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:12:54.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:12:54.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:12:54.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:12:54.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:12:54.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:54.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:54.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:12:54.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:12:54.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:12:54.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:12:54.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:12:54.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:12:54.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:12:54.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:12:54.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:12:54.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:12:54.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:12:54.099 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:12:54.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:12:54.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:12:54.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:12:54.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:12:54.105 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:12:54.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:12:54.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:12:54.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:12:54.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:12:54.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:12:54.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:12:54.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:12:54.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:12:54.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:12:54.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:12:54.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:12:54.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:12:54.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:12:54.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:54.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:54.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:12:54.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:12:54.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:12:54.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:12:54.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:12:54.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:54.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:54.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T06:12:54.120 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:12:54.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:12:54.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:12:54.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:12:54.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-02-05T06:12:54.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5931: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-02-05T06:12:54.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-02-05T06:12:54.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5933: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-02-05T06:12:54.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-02-05T06:12:54.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:12:54.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:12:54.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:12:54.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:12:54.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:12:54.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-02-05T06:12:54.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:12:54.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:12:54.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:12:54.501 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:12:54.501 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:54.501 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:54.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:12:54.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T06:12:54.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:12:54.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:12:54.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:12:54.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:12:54.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:12:54.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:12:54.748 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:12:54.748 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:12:54.748 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:12:54.749 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:12:54.749 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:54.749 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:54.749 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:12:54.750 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:12:54.750 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T06:12:54.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:12:54.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:12:54.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:12:54.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:12:54.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:12:54.871 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:12:54.871 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:12:54.871 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:12:54.872 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:12:54.872 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:54.872 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:54.872 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:12:54.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:12:54.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T06:12:54.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-02-05T06:12:54.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:12:54.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:12:54.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:12:54.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:12:54.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:12:54.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:12:55.914 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:12:55.914 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:12:55.914 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:12:55.915 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:12:55.915 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:55.915 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:55.915 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:12:55.916 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:12:55.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:12:55.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5937: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-02-05T06:12:55.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:12:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:12:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:12:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:12:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:12:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:12:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:12:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:12:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:12:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:12:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:12:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:12:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:12:55.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:12:55.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:12:55.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:12:55.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:12:55.954 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:12:55.954 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:55.954 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:12:55.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:12:55.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:12:55.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:12:55.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:12:55.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:12:55.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:12:55.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:12:55.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:12:55.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:12:55.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:12:55.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:12:55.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:12:55.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:12:55.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:12:55.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:12:55.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=54b8b6e8-db30-49a2-a84e-c5cfda1f8c03 2024-02-05T06:12:55.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 54b8b6e8-db30-49a2-a84e-c5cfda1f8c03' 2024-02-05T06:12:55.957 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 54b8b6e8-db30-49a2-a84e-c5cfda1f8c03 2024-02-05T06:12:55.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:12:55.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBofMBlhABAABAAtcc9FixjLpdR8IElKXYPRg== 2024-02-05T06:12:55.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBofMBlhABAABAAtcc9FixjLpdR8IElKXYPRg=="}' 2024-02-05T06:12:55.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 54b8b6e8-db30-49a2-a84e-c5cfda1f8c03 -i td/osd-scrub-repair/0/new.json 2024-02-05T06:12:56.290 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:12:56.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:12:56.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBofMBlhABAABAAtcc9FixjLpdR8IElKXYPRg== --osd-uuid 54b8b6e8-db30-49a2-a84e-c5cfda1f8c03 2024-02-05T06:12:56.343 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:56.371+0000 7fbf50631640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:56.374 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:56.399+0000 7fbf50631640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:56.375 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:56.403+0000 7fbf50631640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:56.376 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:56.403+0000 7fbf50631640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:12:56.376 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:56.403+0000 7fbf50631640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:12:56.376 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:56.403+0000 7fbf50631640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:12:56.483 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:12:56.507+0000 7fbf50631640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:13:04.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:13:04.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:13:04.462 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:13:04.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:13:04.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:13:04.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:13:04.849 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:13:04.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:13:04.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:13:04.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:13:04.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:13:04.882 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:04.907+0000 7f4cca19d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:04.932 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:04.959+0000 7f4cca19d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:04.933 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:04.959+0000 7f4cca19d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:05.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:13:05.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:13:05.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:13:05.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:13:05.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:13:05.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:05.159 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:13:05.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:13:05.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:05.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:05.509 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:05.535+0000 7f4cca19d640 -1 Falling back to public interface 2024-02-05T06:13:05.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:06.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:06.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:06.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:13:06.516 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:13:06.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:06.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:06.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:07.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:07.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:07.826 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:13:07.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:13:07.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:07.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:08.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:08.487 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:08.515+0000 7f4cca19d640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:13:09.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:09.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:09.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:13:09.125 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:13:09.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:09.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:09.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:10.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:10.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:10.659 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:13:10.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:13:10.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:10.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:11.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:12.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:12.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:12.169 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:13:12.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:13:12.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:12.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:12.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:13.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:13.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:13.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:13:13.669 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:13:13.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:13.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:13.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:14.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:14.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:14.981 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:13:14.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:13:14.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:14.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:15.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:16.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:16.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:16.385 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:13:16.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:13:16.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:16.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:16.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:17.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:17.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:17.838 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:13:17.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:13:17.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:17.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:18.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:19.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:19.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:19.150 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:13:19.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:13:19.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:19.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:19.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:20.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:20.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:20.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:13:20.568 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:13:20.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:20.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:21.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:22.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:22.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:22.075 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:13:22.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:13:22.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:22.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:22.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:23.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:23.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:23.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:13:23.376 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:13:23.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:23.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:23.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:24.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:24.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:24.894 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:13:24.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:13:24.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:24.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:25.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:26.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:26.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:26.215 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:13:26.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:13:26.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:26.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:26.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:27.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:27.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:27.728 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:13:27.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:13:27.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:27.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:28.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:29.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:29.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:29.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:13:29.049 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:13:29.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:29.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:30.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:31.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:31.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:31.081 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:13:31.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:13:31.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:31.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:32.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:33.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:33.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:33.060 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:13:33.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:13:33.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:33.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:34.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:35.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:35.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:35.068 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:13:35.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:13:35.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:35.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:35.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:36.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:36.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:36.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T06:13:36.461 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T06:13:36.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:36.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:37.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:38.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:38.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:38.455 INFO:tasks.workunit.client.0.smithi148.stdout:22 2024-02-05T06:13:38.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-02-05T06:13:38.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:38.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:38.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:39.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:39.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:39.770 INFO:tasks.workunit.client.0.smithi148.stdout:23 2024-02-05T06:13:39.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-02-05T06:13:39.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:39.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:40.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:41.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:41.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:41.396 INFO:tasks.workunit.client.0.smithi148.stdout:24 2024-02-05T06:13:41.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-02-05T06:13:41.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:41.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:41.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:42.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:42.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:42.720 INFO:tasks.workunit.client.0.smithi148.stdout:25 2024-02-05T06:13:42.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-02-05T06:13:42.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:42.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:43.066 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3823387783,v1:127.0.0.1:6803/3823387783] [v2:127.0.0.1:6804/3823387783,v1:127.0.0.1:6805/3823387783] exists,up 54b8b6e8-db30-49a2-a84e-c5cfda1f8c03 2024-02-05T06:13:43.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:13:43.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:13:43.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:13:43.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-02-05T06:13:43.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:13:43.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:13:43.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:13:43.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:13:43.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:13:43.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:13:43.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:13:43.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:13:43.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:13:43.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:13:43.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:13:43.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:13:43.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:13:43.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:13:43.071 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:13:43.071 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:13:43.071 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:13:43.071 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:13:43.071 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:13:43.072 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:13:43.072 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:13:43.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:13:43.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:13:43.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:13:43.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:13:43.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:13:43.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:13:43.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:13:43.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:13:43.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:13:43.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:13:43.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:13:43.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:13:43.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:13:43.073 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:13:43.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=69a64e49-f6c4-4128-b9cf-14e47cb6cb93 2024-02-05T06:13:43.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 69a64e49-f6c4-4128-b9cf-14e47cb6cb93' 2024-02-05T06:13:43.074 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 69a64e49-f6c4-4128-b9cf-14e47cb6cb93 2024-02-05T06:13:43.075 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:13:43.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXfMBlUKWcBxAAfaumK/gni2VRPLBoGZOpew== 2024-02-05T06:13:43.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXfMBlUKWcBxAAfaumK/gni2VRPLBoGZOpew=="}' 2024-02-05T06:13:43.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 69a64e49-f6c4-4128-b9cf-14e47cb6cb93 -i td/osd-scrub-repair/1/new.json 2024-02-05T06:13:43.694 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:13:43.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T06:13:43.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCXfMBlUKWcBxAAfaumK/gni2VRPLBoGZOpew== --osd-uuid 69a64e49-f6c4-4128-b9cf-14e47cb6cb93 2024-02-05T06:13:43.753 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:43.778+0000 7fa423086640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:43.829 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:43.854+0000 7fa423086640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:43.831 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:43.858+0000 7fa423086640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:43.832 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:43.858+0000 7fa423086640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:13:43.832 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:43.858+0000 7fa423086640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:13:43.832 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:43.858+0000 7fa423086640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:13:43.918 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:43.942+0000 7fa423086640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T06:13:52.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T06:13:52.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:13:52.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:13:52.351 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:13:52.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:13:52.756 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:13:52.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:13:52.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:13:52.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:13:52.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:13:52.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:13:52.791 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:52.822+0000 7f776b406640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:52.835 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:52.866+0000 7f776b406640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:52.837 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:52.866+0000 7f776b406640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:53.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:13:53.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:13:53.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:13:53.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:13:53.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:13:53.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:53.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:13:53.074 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:13:53.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:53.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:53.396 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:53.426+0000 7f776b406640 -1 Falling back to public interface 2024-02-05T06:13:53.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:54.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:54.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:54.406 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:13:54.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:13:54.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:54.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:54.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:55.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:55.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:55.727 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:13:55.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:13:55.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:55.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:56.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:56.416 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:13:56.446+0000 7f776b406640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:13:57.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:57.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:57.047 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:13:57.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:13:57.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:57.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:57.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:58.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:58.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:58.644 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:13:58.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:13:58.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:58.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:59.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:00.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:00.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:00.113 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:14:00.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:14:00.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:00.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:00.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:01.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:01.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:01.423 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:14:01.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:14:01.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:01.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:01.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:02.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:02.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:02.730 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:14:02.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:14:02.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:02.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:03.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:04.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:04.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:04.294 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:14:04.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:14:04.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:04.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:04.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:05.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:05.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:05.950 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:14:05.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:14:05.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:05.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:06.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:07.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:07.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:07.307 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:14:07.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:14:07.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:07.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:07.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:08.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:08.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:08.634 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:14:08.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:14:08.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:08.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:09.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:10.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:10.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:10.186 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:14:10.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:14:10.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:10.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:10.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:11.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:11.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:11.554 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:14:11.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:14:11.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:11.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:12.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:13.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:13.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:13.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:14:13.064 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:14:13.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:13.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:13.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:14.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:14.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:14.465 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:14:14.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:14:14.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:14.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:14.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:15.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:15.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:15.931 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:14:15.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:14:15.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:15.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:16.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:17.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:17.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:17.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:14:17.249 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:14:17.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:17.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:18.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:19.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:19.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:19.033 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:14:19.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:14:19.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:19.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:19.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:20.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:20.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:20.436 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:14:20.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:14:20.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:20.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:21.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:22.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:22.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:22.480 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:14:22.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:14:22.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:22.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:23.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:24.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:24.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:24.550 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T06:14:24.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T06:14:24.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:24.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:25.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:26.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:26.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:26.662 INFO:tasks.workunit.client.0.smithi148.stdout:22 2024-02-05T06:14:26.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-02-05T06:14:26.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:26.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:27.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:28.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:28.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:28.453 INFO:tasks.workunit.client.0.smithi148.stdout:23 2024-02-05T06:14:28.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-02-05T06:14:28.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:28.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:29.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:30.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:30.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:30.003 INFO:tasks.workunit.client.0.smithi148.stdout:24 2024-02-05T06:14:30.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-02-05T06:14:30.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:30.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:30.321 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3477663011,v1:127.0.0.1:6811/3477663011] [v2:127.0.0.1:6812/3477663011,v1:127.0.0.1:6813/3477663011] exists,up 69a64e49-f6c4-4128-b9cf-14e47cb6cb93 2024-02-05T06:14:30.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:14:30.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:14:30.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:14:30.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-02-05T06:14:30.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T06:14:30.884 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T06:14:30.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T06:14:30.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T06:14:31.526 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T06:14:31.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:14:32.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T06:14:35.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-02-05T06:14:35.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:14:35.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:14:35.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:14:35.164 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:14:35.164 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:14:35.164 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:14:35.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:14:35.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:14:35.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:14:35.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:14:35.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:14:35.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:14:35.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:14:35.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:14:35.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:14:35.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:14:35.610 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:14:35.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:14:35.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:14:35.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:14:35.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-02-05T06:14:35.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-02-05T06:14:35.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-02-05T06:14:35.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:14:35.826 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:14:35.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-02-05T06:14:35.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-02-05T06:14:35.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672966' 2024-02-05T06:14:35.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:14:35.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-02-05T06:14:35.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:14:35.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:14:35.973 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-02-05T06:14:35.973 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:14:35.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-02-05T06:14:35.975 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836497 2024-02-05T06:14:35.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-02-05T06:14:35.975 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:14:36.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-02-05T06:14:36.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:14:37.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:14:37.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:14:37.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-02-05T06:14:37.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:14:37.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-02-05T06:14:37.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:14:37.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:14:37.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-02-05T06:14:37.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:14:37.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-02-05T06:14:37.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-02-05T06:14:37.685 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672966 2024-02-05T06:14:37.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:14:38.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672966 2024-02-05T06:14:38.091 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:14:38.092 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:14:38.092 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:14:38.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T06:14:38.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:14:38.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:14:38.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:14:38.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:14:38.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:14:38.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:14:38.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:14:38.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:14:38.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:14:38.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:14:38.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:14:39.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T06:14:39.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:14:39.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:14:39.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-02-05T06:14:39.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-02-05T06:14:39.581 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'foo' created 2024-02-05T06:14:39.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:14:40.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-02-05T06:14:40.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-02-05T06:14:41.563 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'csr_pool' created 2024-02-05T06:14:41.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:14:42.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5944: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-02-05T06:14:42.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:14:42.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:14:42.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:14:42.588 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:14:42.589 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:14:42.589 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:14:42.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:14:42.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:14:42.590 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:14:42.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:14:42.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:14:42.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:14:42.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:14:42.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:14:42.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:14:43.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:14:43.077 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:14:43.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:14:43.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:14:43.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:14:43.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-02-05T06:14:43.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-02-05T06:14:43.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-02-05T06:14:43.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:14:43.310 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:14:43.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-02-05T06:14:43.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-02-05T06:14:43.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672968' 2024-02-05T06:14:43.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:14:43.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-02-05T06:14:43.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:14:43.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:14:43.461 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-02-05T06:14:43.462 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:14:43.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-02-05T06:14:43.463 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836500 2024-02-05T06:14:43.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-02-05T06:14:43.464 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:14:43.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836500 2024-02-05T06:14:43.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:14:44.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:14:44.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:14:45.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-02-05T06:14:45.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:14:45.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-02-05T06:14:45.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:14:45.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:14:45.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-02-05T06:14:45.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:14:45.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-02-05T06:14:45.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-02-05T06:14:45.194 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672968 2024-02-05T06:14:45.194 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:14:45.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-02-05T06:14:45.537 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:14:45.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:14:45.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:14:45.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:14:45.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:14:45.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:14:45.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:14:45.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:14:45.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:14:45.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:14:45.890 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:14:46.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:14:46.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:14:46.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:14:46.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:14:46.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-02-05T06:14:46.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-02-05T06:14:46.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-02-05T06:14:46.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-02-05T06:14:46.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:14:46.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:14:46.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-02-05T06:14:46.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:14:46.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:14:46.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:14:47.017 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:14:47.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:14:47.536 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:14:47.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:14:47.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:14:47.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:14:47.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-02-05T06:14:48.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-02-05T06:14:48.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-02-05T06:14:48.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-02-05T06:14:48.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-02-05T06:14:48.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:14:48.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T06:14:48.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-02-05T06:14:48.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:14:48.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:14:48.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:14:48.857 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:14:48.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:14:49.383 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:14:49.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:14:49.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:14:49.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:14:49.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-02-05T06:14:50.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-02-05T06:14:50.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-02-05T06:14:50.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-02-05T06:14:50.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-02-05T06:14:50.134 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-02-05T06:14:50.134 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:14:50.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-02-05T06:14:50.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-02-05T06:14:50.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-02-05T06:14:50.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-02-05T06:14:50.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:14:50.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-02-05T06:14:50.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-02-05T06:14:50.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-02-05T06:14:51.164 INFO:tasks.workunit.client.0.smithi148.stdout:created pool csr_pool snap snap1 2024-02-05T06:14:51.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-02-05T06:14:51.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-02-05T06:14:51.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-02-05T06:14:51.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-02-05T06:14:51.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-02-05T06:14:51.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-02-05T06:14:51.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-02-05T06:14:51.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5968: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-02-05T06:14:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-02-05T06:14:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:14:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:14:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T06:14:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:14:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-02-05T06:14:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:14:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:14:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T06:14:51.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:14:51.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:14:51.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:14:51.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:14:51.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:14:51.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:14:51.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:14:51.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:14:51.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-02-05T06:14:51.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:14:51.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:14:51.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:14:51.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:14:51.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:14:51.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-02-05T06:14:54.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T06:14:54.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:14:54.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:14:54.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:14:54.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:14:54.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:14:54.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:14:54.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:14:54.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:14:54.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:14:54.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:14:54.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:14:54.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:14:54.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:14:54.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:14:54.654 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:14:54.654 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:14:54.654 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:14:54.658 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:14:54.658 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:14:54.658 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:14:54.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:14:54.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:14:54.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:14:54.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:14:54.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:14:54.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:14:54.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:14:54.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:14:54.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:14:54.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:14:54.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:14:54.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:14:54.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:14:54.660 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:14:54.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:14:54.660 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:14:54.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:14:54.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:14:54.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:14:54.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:14:54.691 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:14:54.720+0000 7fb9845d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:14:54.693 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:14:54.720+0000 7fb9845d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:14:54.695 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:14:54.724+0000 7fb9845d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:14:54.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:14:54.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:14:54.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:14:54.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:14:54.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:14:54.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:54.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:14:54.972 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:14:54.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:54.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:55.241 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:14:55.268+0000 7fb9845d6640 -1 Falling back to public interface 2024-02-05T06:14:55.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:56.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:56.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:56.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:14:56.281 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:14:56.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:56.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:56.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:57.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:57.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:57.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:14:57.595 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:14:57.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:57.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:57.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:58.295 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:14:58.324+0000 7fb9845d6640 -1 osd.1 30 log_to_monitors true 2024-02-05T06:14:58.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:58.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:58.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:14:58.985 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:14:58.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:58.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:59.425 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 34 up_thru 23 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/4219415482,v1:127.0.0.1:6811/4219415482] [v2:127.0.0.1:6812/4219415482,v1:127.0.0.1:6813/4219415482] exists,up 69a64e49-f6c4-4128-b9cf-14e47cb6cb93 2024-02-05T06:14:59.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:14:59.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:14:59.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:14:59.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:14:59.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:14:59.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:14:59.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:14:59.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:14:59.426 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:14:59.426 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:14:59.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:14:59.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:14:59.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:14:59.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:14:59.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:14:59.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:14:59.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:14:59.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:14:59.556 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:15:00.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:15:00.271 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:15:00.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:15:00.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:15:00.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:15:00.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-02-05T06:15:00.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-02-05T06:15:00.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-02-05T06:15:00.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:15:00.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:15:00.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-02-05T06:15:00.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-02-05T06:15:00.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-146028888066' 2024-02-05T06:15:00.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:15:00.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-02-05T06:15:00.828 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:15:00.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:15:00.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-02-05T06:15:00.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:15:00.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-02-05T06:15:00.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-02-05T06:15:00.833 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 21474836504 2024-02-05T06:15:00.834 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:15:01.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-02-05T06:15:01.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:15:02.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:15:02.176 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:15:02.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-02-05T06:15:02.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:15:03.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:15:03.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:15:03.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-02-05T06:15:03.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:15:03.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-146028888066 2024-02-05T06:15:03.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:15:03.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:15:03.826 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-146028888066 2024-02-05T06:15:03.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:15:03.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-02-05T06:15:03.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 146028888066' 2024-02-05T06:15:03.827 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 146028888066 2024-02-05T06:15:03.828 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:15:04.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888066 2024-02-05T06:15:04.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:15:04.204 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:15:04.204 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:15:04.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:15:04.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:15:04.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:15:04.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:15:04.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:15:04.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:15:04.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:15:04.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:15:04.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:15:04.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:15:04.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:15:04.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:15:05.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:15:05.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:15:05.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:15:05.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-02-05T06:15:05.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-02-05T06:15:05.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-02-05T06:15:05.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-02-05T06:15:05.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-02-05T06:15:05.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5973: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-02-05T06:15:05.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5974: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-02-05T06:15:05.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:15:05.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:15:05.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:15:05.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:15:05.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-02-05T06:15:05.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:15:05.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:15:05.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:15:05.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:15:05.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:15:05.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:15:05.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:15:05.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:15:05.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:15:05.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:15:05.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:15:05.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-02-05T06:15:05.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:15:05.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:15:05.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:15:05.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:15:05.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:15:05.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-02-05T06:15:08.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:15:08.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:15:08.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:15:08.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:15:08.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:15:08.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:15:08.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:15:08.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:15:08.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:15:08.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:15:08.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:15:08.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:15:08.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:15:08.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:15:08.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:15:08.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:15:08.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:15:08.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:15:08.715 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:15:08.716 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:08.716 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:15:08.716 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:15:08.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:15:08.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:15:08.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:15:08.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:15:08.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:15:08.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:15:08.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:15:08.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:15:08.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:15:08.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:15:08.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:15:08.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:15:08.717 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:15:08.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:15:08.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:15:08.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:15:08.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:15:08.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:15:08.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:15:08.751 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:08.780+0000 7f24aa8e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:08.754 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:08.784+0000 7f24aa8e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:08.756 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:08.784+0000 7f24aa8e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:09.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:15:09.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:15:09.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:15:09.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:15:09.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:15:09.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:09.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:15:09.036 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:15:09.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:09.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:09.288 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:09.316+0000 7f24aa8e2640 -1 Falling back to public interface 2024-02-05T06:15:09.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:10.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:10.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:10.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:15:10.366 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:15:10.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:10.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:10.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:11.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:11.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:11.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:15:11.725 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:15:11.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:11.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:12.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:12.441 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:12.468+0000 7f24aa8e2640 -1 osd.0 36 log_to_monitors true 2024-02-05T06:15:13.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:13.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:13.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:15:13.044 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:15:13.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:13.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:13.626 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 40 up_thru 34 down_at 37 last_clean_interval [5,36) [v2:127.0.0.1:6802/341680647,v1:127.0.0.1:6803/341680647] [v2:127.0.0.1:6804/341680647,v1:127.0.0.1:6805/341680647] exists,up 54b8b6e8-db30-49a2-a84e-c5cfda1f8c03 2024-02-05T06:15:13.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:15:13.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:15:13.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:15:13.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:15:13.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:15:13.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:15:13.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:15:13.627 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:15:13.628 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:15:13.628 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:15:13.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:15:13.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:15:13.629 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:15:13.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:15:13.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:15:13.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:15:13.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:15:13.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:15:13.756 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:15:14.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:15:14.070 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:15:14.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:15:14.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:15:14.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:15:14.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-02-05T06:15:14.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-02-05T06:15:14.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842' 2024-02-05T06:15:14.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:15:14.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:15:14.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888070 2024-02-05T06:15:14.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888070 2024-02-05T06:15:14.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-146028888070' 2024-02-05T06:15:14.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:15:14.421 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:15:14.421 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691842 2024-02-05T06:15:14.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:15:14.423 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691842 2024-02-05T06:15:14.423 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:15:14.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-02-05T06:15:14.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-02-05T06:15:14.425 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 171798691842 2024-02-05T06:15:14.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:15:14.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-02-05T06:15:14.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:15:14.738 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-146028888070 2024-02-05T06:15:14.738 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:15:14.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:15:14.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-146028888070 2024-02-05T06:15:14.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:15:14.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888070 2024-02-05T06:15:14.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 146028888070' 2024-02-05T06:15:14.744 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 146028888070 2024-02-05T06:15:14.744 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:15:15.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888070 -lt 146028888070 2024-02-05T06:15:15.092 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:15:15.092 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:15:15.092 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:15:15.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:15:15.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:15:15.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:15:15.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:15:15.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:15:15.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:15:15.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:15:15.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:15:15.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:15:15.756 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:15:15.756 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:15:15.757 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:15:16.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T06:15:16.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-02-05T06:15:16.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:15:16.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T06:15:16.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:15:16.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:15:16.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:15:16.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:15:16.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:15:16.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:15:16.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:15:16.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:15:16.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:15:16.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:15:16.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:15:16.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:15:16.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:15:16.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:15:16.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T06:15:16.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T06:15:16.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:15:16.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:15:16.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:15:16.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:15:16.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:15:16.938 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:15:16.938 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:15:17.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:15:17.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:15:17.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:15:17.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:15:17.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:15:17.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:15:17.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:15:17.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:15:17.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:15:17.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:15:17.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:15:17.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:15:17.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:15:17.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:15:17.864 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:15:17.864 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:15:17.864 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:15:18.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T06:15:18.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T06:15:18.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:15:18.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:15:18.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:15:18.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:15:18.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:15:18.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:15:18.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:15:18.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:15:18.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:15:18.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T06:15:18.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:15:18.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T06:15:19.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:15:19.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:15:19.019 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:15:19.020 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:15:19.020 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:15:19.020 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:15:19.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:15:19.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:15:19.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:15:19.356 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:15:19.356 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:15:19.356 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:15:19.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:15:19.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:15:19.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:15:19.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5979: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-02-05T06:15:19.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-02-05T06:15:19.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-02-05T06:15:19.710 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-02-05T06:15:19.710 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:15:19.710 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:15:19.711 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:15:19.711 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:15:20.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-02-05T06:14:41.394013+0000 2024-02-05T06:15:20.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-02-05T06:15:20.352 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 3.0 on osd.1 to scrub 2024-02-05T06:15:20.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-02-05T06:14:41.394013+0000 2024-02-05T06:15:20.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T06:15:20.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:14:41.394013+0000 2024-02-05T06:15:20.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:15:20.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:15:20.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:15:20.370 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:15:20.371 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:15:20.371 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:15:20.371 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:15:20.371 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:15:20.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:14:41.394013+0000 '>' 2024-02-05T06:14:41.394013+0000 2024-02-05T06:15:20.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:15:21.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:15:21.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:15:21.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:15:21.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:15:21.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:15:21.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:15:21.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:15:21.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:14:41.394013+0000 '>' 2024-02-05T06:14:41.394013+0000 2024-02-05T06:15:21.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:15:22.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:15:22.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:15:22.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:15:22.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:15:22.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:15:23.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:15:23.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:15:23.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:14:41.394013+0000 '>' 2024-02-05T06:14:41.394013+0000 2024-02-05T06:15:23.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:15:24.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:15:24.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:15:24.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T06:15:24.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T06:15:24.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:15:24.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:15:24.352 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T06:15:24.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:15:20.706932+0000 '>' 2024-02-05T06:14:41.394013+0000 2024-02-05T06:15:24.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:15:24.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5983: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-02-05T06:15:24.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-02-05T06:15:24.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-02-05T06:15:24.725 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-02-05T06:15:24.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-02-05T06:15:24.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5989: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-02-05T06:15:24.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-02-05T06:15:24.781 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T06:15:24.781 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T06:15:24.781 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T06:15:24.781 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T06:15:24.781 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T06:15:24.781 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T06:15:24.781 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T06:15:24.781 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T06:15:24.781 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T06:15:24.781 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T06:15:24.781 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T06:15:24.781 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T06:15:24.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-02-05T06:15:24.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T06:15:24.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-02-05T06:15:24.814 INFO:tasks.workunit.client.0.smithi148.stderr: . as $in 2024-02-05T06:15:24.814 INFO:tasks.workunit.client.0.smithi148.stderr: | if type == "object" then 2024-02-05T06:15:24.814 INFO:tasks.workunit.client.0.smithi148.stderr: reduce keys[] as $key 2024-02-05T06:15:24.814 INFO:tasks.workunit.client.0.smithi148.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T06:15:24.814 INFO:tasks.workunit.client.0.smithi148.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T06:15:24.814 INFO:tasks.workunit.client.0.smithi148.stderr: else f 2024-02-05T06:15:24.814 INFO:tasks.workunit.client.0.smithi148.stderr: end; 2024-02-05T06:15:24.814 INFO:tasks.workunit.client.0.smithi148.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T06:15:24.814 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T06:15:24.814 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T06:15:24.814 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T06:15:24.814 INFO:tasks.workunit.client.0.smithi148.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-02-05T06:15:24.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-02-05T06:15:24.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T06:15:24.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T06:15:24.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T06:15:24.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6159: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-02-05T06:15:24.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6164: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-02-05T06:15:24.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-02-05T06:15:24.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6170: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-02-05T06:15:24.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6171: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-02-05T06:15:24.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-02-05T06:15:24.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-02-05T06:15:24.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-02-05T06:15:24.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-02-05T06:15:24.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-02-05T06:15:24.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-02-05T06:15:24.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-02-05T06:15:24.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-02-05T06:15:24.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-02-05T06:15:24.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-02-05T06:15:24.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-02-05T06:15:24.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-02-05T06:15:24.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-02-05T06:15:24.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-02-05T06:15:24.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-02-05T06:15:24.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-02-05T06:15:24.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-02-05T06:15:24.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6187: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-02-05T06:15:24.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6193: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-02-05T06:15:25.448 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'csr_pool' removed 2024-02-05T06:15:25.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:15:25.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:15:25.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:15:25.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:15:25.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:15:25.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:15:25.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:15:25.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:15:25.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:15:25.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:15:25.783 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:15:25.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:15:25.786 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:15:25.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:15:25.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:15:25.788 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:15:25.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:15:25.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:15:25.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:15:25.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:15:25.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:15:25.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:15:25.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:15:25.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:15:25.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:25.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:15:25.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:15:25.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:15:25.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:15:25.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:15:25.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:15:25.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:15:25.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:15:25.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:15:25.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:15:25.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:15:25.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:15:25.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:15:25.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:15:25.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:15:25.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:15:25.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:15:25.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:15:25.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:15:25.821 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:15:25.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:15:25.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:15:25.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:15:25.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:15:25.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:15:25.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:15:25.826 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:15:25.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:15:25.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:15:25.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:15:25.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:15:25.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:25.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:15:25.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:15:25.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:15:25.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:15:25.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:15:25.834 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:15:25.834 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:25.834 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:15:25.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T06:15:25.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:15:25.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:15:25.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:15:25.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:15:25.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-02-05T06:15:25.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-02-05T06:15:25.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T06:15:25.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-02-05T06:15:25.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-02-05T06:15:25.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-02-05T06:15:25.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:15:25.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:15:25.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:15:25.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:15:25.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:15:25.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T06:15:26.292 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:15:26.292 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:15:26.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:15:26.293 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:15:26.293 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:26.293 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:15:26.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:15:26.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T06:15:26.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:15:26.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:15:26.548 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:15:26.548 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:15:26.548 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:15:26.548 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:15:26.548 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:15:26.549 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:15:26.549 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:15:26.549 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:15:26.549 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:26.550 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:15:26.550 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:15:26.550 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:15:26.550 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T06:15:26.666 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:15:26.666 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:15:26.667 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:15:26.667 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:15:26.667 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:15:26.668 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:15:26.668 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:15:26.668 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:15:26.668 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:15:26.668 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:26.668 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:15:26.669 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:15:26.669 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:15:26.669 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T06:15:26.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-02-05T06:15:26.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:15:26.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:15:26.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:15:26.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:15:26.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:15:26.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:15:27.679 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:15:27.679 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:15:27.679 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:15:27.679 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:15:27.679 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:27.679 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:15:27.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:15:27.681 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:15:27.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:15:27.716 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-02-05T06:15:27.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:15:27.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-02-05T06:15:27.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:15:27.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:15:27.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:15:27.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:15:27.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:15:27.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:15:27.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:15:27.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:15:27.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:15:27.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:15:27.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:15:27.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:15:27.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:15:27.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:15:27.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:15:27.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:15:27.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:15:27.720 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:15:27.720 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:27.720 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:15:27.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:15:27.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:15:27.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:15:27.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:15:27.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:15:27.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:15:27.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:15:27.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:15:27.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:15:27.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:15:27.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:15:27.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:15:27.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:15:27.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:15:27.723 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:15:27.723 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 b626794e-171b-49c5-aadf-54d16e8cc9b1 2024-02-05T06:15:27.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b626794e-171b-49c5-aadf-54d16e8cc9b1 2024-02-05T06:15:27.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b626794e-171b-49c5-aadf-54d16e8cc9b1' 2024-02-05T06:15:27.723 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:15:27.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/fMBlgk0GLhAAb4SxPzRppdWtSonSIdPtIA== 2024-02-05T06:15:27.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/fMBlgk0GLhAAb4SxPzRppdWtSonSIdPtIA=="}' 2024-02-05T06:15:27.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b626794e-171b-49c5-aadf-54d16e8cc9b1 -i td/osd-scrub-repair/0/new.json 2024-02-05T06:15:28.023 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:15:28.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:15:28.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD/fMBlgk0GLhAAb4SxPzRppdWtSonSIdPtIA== --osd-uuid b626794e-171b-49c5-aadf-54d16e8cc9b1 2024-02-05T06:15:28.079 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:28.108+0000 7fef9f0fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:28.108 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:28.136+0000 7fef9f0fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:28.109 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:28.136+0000 7fef9f0fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:28.110 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:28.140+0000 7fef9f0fe640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:15:28.110 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:28.140+0000 7fef9f0fe640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:15:28.110 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:28.140+0000 7fef9f0fe640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:15:28.216 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:28.244+0000 7fef9f0fe640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:15:36.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:15:36.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:15:36.441 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:15:36.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:15:36.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:15:36.835 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:15:36.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:15:36.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:15:36.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:15:36.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:15:36.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:15:36.868 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:36.895+0000 7f2a2af1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:36.916 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:36.943+0000 7f2a2af1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:36.918 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:36.947+0000 7f2a2af1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:37.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:15:37.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:15:37.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:15:37.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:15:37.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:15:37.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:37.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:15:37.146 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:15:37.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:37.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:37.483 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:37.511+0000 7f2a2af1a640 -1 Falling back to public interface 2024-02-05T06:15:37.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:38.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:38.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:38.506 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:15:38.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:15:38.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:38.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:38.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:39.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:39.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:39.822 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:15:39.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:15:39.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:39.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:40.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:40.428 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:15:40.455+0000 7f2a2af1a640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:15:41.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:41.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:41.126 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:15:41.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:15:41.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:41.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:41.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:42.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:42.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:42.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:15:42.582 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:15:42.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:42.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:42.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:43.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:43.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:43.953 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:15:43.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:15:43.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:43.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:44.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:45.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:45.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:45.307 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:15:45.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:15:45.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:45.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:45.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:46.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:46.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:46.781 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:15:46.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:15:46.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:46.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:47.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:48.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:48.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:48.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:15:48.100 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:15:48.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:48.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:48.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:49.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:49.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:49.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:15:49.547 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:15:49.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:49.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:49.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:50.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:50.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:50.859 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:15:50.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:15:50.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:50.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:51.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:52.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:52.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:52.162 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:15:52.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:15:52.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:52.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:52.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:53.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:53.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:53.656 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:15:53.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:15:53.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:53.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:53.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:54.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:54.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:54.965 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:15:54.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:15:54.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:54.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:55.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:56.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:56.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:56.391 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:15:56.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:15:56.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:56.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:56.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:57.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:57.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:57.914 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:15:57.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:15:57.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:57.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:58.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:59.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:59.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:59.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:15:59.374 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:15:59.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:59.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:59.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:00.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:00.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:00.720 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:16:00.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:16:00.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:00.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:16:01.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:02.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:02.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:02.257 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:16:02.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:16:02.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:02.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:16:03.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:04.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:04.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:04.341 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:16:04.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:16:04.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:04.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:16:05.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:06.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:06.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:06.618 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:16:06.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:16:06.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:06.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:16:07.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:08.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:08.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:08.322 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T06:16:08.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T06:16:08.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:08.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:16:09.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:10.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:10.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:10.175 INFO:tasks.workunit.client.0.smithi148.stdout:22 2024-02-05T06:16:10.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-02-05T06:16:10.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:10.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:16:11.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:12.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:12.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:12.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-02-05T06:16:12.199 INFO:tasks.workunit.client.0.smithi148.stdout:23 2024-02-05T06:16:12.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:12.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:16:12.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:13.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:13.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:13.833 INFO:tasks.workunit.client.0.smithi148.stdout:24 2024-02-05T06:16:13.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-02-05T06:16:13.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:13.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:16:14.139 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1661563287,v1:127.0.0.1:6803/1661563287] [v2:127.0.0.1:6804/1661563287,v1:127.0.0.1:6805/1661563287] exists,up b626794e-171b-49c5-aadf-54d16e8cc9b1 2024-02-05T06:16:14.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:16:14.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:16:14.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:16:14.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:16:14.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-02-05T06:16:14.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:16:14.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:16:14.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:16:14.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:16:14.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:16:14.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:16:14.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:16:14.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:16:14.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:16:14.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:16:14.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:16:14.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:16:14.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:16:14.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:16:14.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:16:14.145 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:16:14.145 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:16:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:16:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:16:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:16:14.145 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:16:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:16:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:16:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:16:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:16:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:16:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:16:14.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:16:14.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:16:14.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:16:14.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:16:14.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:16:14.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:16:14.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:16:14.146 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:16:14.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=efe4b562-1466-4b99-8b08-f5760603f851 2024-02-05T06:16:14.147 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 efe4b562-1466-4b99-8b08-f5760603f851 2024-02-05T06:16:14.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 efe4b562-1466-4b99-8b08-f5760603f851' 2024-02-05T06:16:14.147 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:16:14.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAufcBlwQvwCxAAGuIQY0RedBE6ok51ud5RPA== 2024-02-05T06:16:14.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAufcBlwQvwCxAAGuIQY0RedBE6ok51ud5RPA=="}' 2024-02-05T06:16:14.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new efe4b562-1466-4b99-8b08-f5760603f851 -i td/osd-scrub-repair/1/new.json 2024-02-05T06:16:14.794 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:16:14.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T06:16:14.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAufcBlwQvwCxAAGuIQY0RedBE6ok51ud5RPA== --osd-uuid efe4b562-1466-4b99-8b08-f5760603f851 2024-02-05T06:16:14.854 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:16:14.883+0000 7f8a834aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:16:14.929 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:16:14.959+0000 7f8a834aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:16:14.931 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:16:14.959+0000 7f8a834aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:16:14.932 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:16:14.959+0000 7f8a834aa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:16:14.932 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:16:14.959+0000 7f8a834aa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:16:14.932 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:16:14.959+0000 7f8a834aa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:16:15.017 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:16:15.043+0000 7f8a834aa640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T06:16:23.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T06:16:23.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:16:23.341 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:16:23.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:16:23.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:16:23.717 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:16:23.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:16:23.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:16:23.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:16:23.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:16:23.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:16:23.743 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:16:23.770+0000 7f14c71ea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:16:23.745 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:16:23.774+0000 7f14c71ea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:16:23.746 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:16:23.774+0000 7f14c71ea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:16:24.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:16:24.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:16:24.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:16:24.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:16:24.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:16:24.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:24.018 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:16:24.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:16:24.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:24.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:24.288 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:16:24.314+0000 7f14c71ea640 -1 Falling back to public interface 2024-02-05T06:16:24.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:25.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:25.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:25.357 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:16:25.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:16:25.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:25.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:25.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:26.676 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:16:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:16:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:27.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:27.384 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:16:27.410+0000 7f14c71ea640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:16:28.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:28.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:28.088 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:16:28.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:16:28.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:28.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:28.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:29.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:29.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:29.758 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:16:29.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:16:29.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:29.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:30.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:31.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:31.078 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:16:31.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:31.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:16:31.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:31.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:31.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:32.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:32.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:32.595 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:16:32.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:16:32.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:32.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:33.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:34.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:34.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:34.017 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:16:34.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:16:34.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:34.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:34.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:35.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:35.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:35.341 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:16:35.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:16:35.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:35.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:35.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:36.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:36.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:36.653 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:16:36.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:16:36.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:36.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:36.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:37.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:37.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:37.973 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:16:37.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:16:37.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:37.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:38.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:39.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:39.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:39.286 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:16:39.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:16:39.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:39.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:39.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:40.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:40.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:40.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:16:40.703 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:16:40.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:40.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:41.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:42.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:42.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:42.110 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:16:42.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:16:42.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:42.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:42.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:43.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:43.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:43.711 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:16:43.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:16:43.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:43.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:44.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:45.022 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:16:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:16:45.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:45.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:45.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:46.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:46.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:46.339 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:16:46.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:16:46.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:46.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:46.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:47.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:47.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:47.897 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:16:47.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:16:47.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:47.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:48.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:49.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:49.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:49.732 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:16:49.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:16:49.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:49.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:50.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:51.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:51.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:51.050 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:16:51.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:16:51.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:51.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:51.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:52.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:52.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:52.633 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:16:52.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:16:52.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:52.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:53.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:54.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:54.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:54.263 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T06:16:54.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T06:16:54.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:54.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:55.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:56.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:56.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:56.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-02-05T06:16:56.313 INFO:tasks.workunit.client.0.smithi148.stdout:22 2024-02-05T06:16:56.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:56.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:57.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:58.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:58.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:58.182 INFO:tasks.workunit.client.0.smithi148.stdout:23 2024-02-05T06:16:58.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-02-05T06:16:58.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:58.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:16:58.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:59.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:59.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:59.746 INFO:tasks.workunit.client.0.smithi148.stdout:24 2024-02-05T06:16:59.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-02-05T06:16:59.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:59.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:17:00.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:01.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:01.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:01.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-02-05T06:17:01.187 INFO:tasks.workunit.client.0.smithi148.stdout:25 2024-02-05T06:17:01.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:01.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:17:01.501 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3735380298,v1:127.0.0.1:6811/3735380298] [v2:127.0.0.1:6812/3735380298,v1:127.0.0.1:6813/3735380298] exists,up efe4b562-1466-4b99-8b08-f5760603f851 2024-02-05T06:17:01.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:17:01.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:17:01.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:17:01.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:17:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-02-05T06:17:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:17:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:17:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T06:17:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:17:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:17:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:17:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:17:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:17:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:17:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:17:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:17:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:17:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:17:01.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:17:01.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:17:01.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:17:01.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:17:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:17:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:17:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:17:01.507 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:17:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:17:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:17:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:17:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:17:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:17:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:17:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:17:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:17:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:17:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:17:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:17:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:17:01.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:17:01.508 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:17:01.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9b6a88b5-69a3-45a1-ae21-0977422f0e62 2024-02-05T06:17:01.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9b6a88b5-69a3-45a1-ae21-0977422f0e62' 2024-02-05T06:17:01.509 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 9b6a88b5-69a3-45a1-ae21-0977422f0e62 2024-02-05T06:17:01.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:17:01.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBdfcBlX9+EIRAAWaqw1p1YYxPft+UUKEnxKA== 2024-02-05T06:17:01.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBdfcBlX9+EIRAAWaqw1p1YYxPft+UUKEnxKA=="}' 2024-02-05T06:17:01.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9b6a88b5-69a3-45a1-ae21-0977422f0e62 -i td/osd-scrub-repair/2/new.json 2024-02-05T06:17:02.059 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:17:02.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T06:17:02.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdfcBlX9+EIRAAWaqw1p1YYxPft+UUKEnxKA== --osd-uuid 9b6a88b5-69a3-45a1-ae21-0977422f0e62 2024-02-05T06:17:02.124 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:17:02.154+0000 7fd9a8da8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:17:02.203 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:17:02.234+0000 7fd9a8da8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:17:02.205 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:17:02.234+0000 7fd9a8da8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:17:02.206 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:17:02.234+0000 7fd9a8da8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:17:02.206 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:17:02.234+0000 7fd9a8da8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:17:02.206 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:17:02.234+0000 7fd9a8da8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:17:02.323 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:17:02.354+0000 7fd9a8da8640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T06:17:10.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T06:17:10.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:17:10.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T06:17:10.631 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T06:17:10.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:17:11.012 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T06:17:11.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T06:17:11.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:17:11.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:17:11.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:17:11.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:17:11.038 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:17:11.065+0000 7f30f92b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:17:11.040 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:17:11.069+0000 7f30f92b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:17:11.041 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:17:11.069+0000 7f30f92b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:17:11.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T06:17:11.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:17:11.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:17:11.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:17:11.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:17:11.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:11.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:17:11.323 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:17:11.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:11.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:11.593 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:17:11.621+0000 7f30f92b1640 -1 Falling back to public interface 2024-02-05T06:17:11.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:12.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:12.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:12.688 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:17:12.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:17:12.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:12.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:13.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:14.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:14.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:14.059 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:17:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:17:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:14.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:14.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:14.525 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:17:14.553+0000 7f30f92b1640 -1 osd.2 0 log_to_monitors true 2024-02-05T06:17:15.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:15.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:15.375 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:17:15.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:17:15.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:15.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:15.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:16.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:16.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:16.823 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:17:16.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:17:16.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:16.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:17.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:18.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:18.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:18.152 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:17:18.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:17:18.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:18.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:18.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:19.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:19.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:19.764 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:17:19.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:17:19.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:19.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:20.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:21.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:21.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:21.156 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:17:21.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:17:21.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:21.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:21.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:22.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:22.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:22.480 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:17:22.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:17:22.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:22.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:22.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:23.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:23.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:23.929 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:17:23.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:17:23.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:23.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:24.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:25.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:25.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:25.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:17:25.234 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:17:25.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:25.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:25.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:26.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:26.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:26.552 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:17:26.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:17:26.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:26.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:26.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:27.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:27.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:27.965 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:17:27.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:17:27.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:27.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:28.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:29.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:29.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:29.347 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:17:29.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:17:29.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:29.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:29.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:30.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:30.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:30.710 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:17:30.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:17:30.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:30.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:31.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:32.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:32.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:32.079 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:17:32.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:17:32.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:32.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:32.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:33.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:33.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:33.386 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:17:33.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:17:33.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:33.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:33.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:34.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:34.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:34.693 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:17:34.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:17:34.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:34.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:35.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:36.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:36.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:36.271 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:17:36.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:17:36.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:36.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:37.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:38.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:38.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:38.246 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:17:38.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:17:38.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:38.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:39.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:40.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:40.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:40.440 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:17:40.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:17:40.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:40.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:41.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:42.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:42.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:42.208 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T06:17:42.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T06:17:42.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:42.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:43.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:44.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:44.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:44.255 INFO:tasks.workunit.client.0.smithi148.stdout:22 2024-02-05T06:17:44.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-02-05T06:17:44.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:44.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:45.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:46.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:46.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:46.205 INFO:tasks.workunit.client.0.smithi148.stdout:23 2024-02-05T06:17:46.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-02-05T06:17:46.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:46.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:47.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:17:48.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:17:48.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:17:48.430 INFO:tasks.workunit.client.0.smithi148.stdout:24 2024-02-05T06:17:48.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-02-05T06:17:48.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:17:48.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:17:48.937 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/40165148,v1:127.0.0.1:6819/40165148] [v2:127.0.0.1:6820/40165148,v1:127.0.0.1:6821/40165148] exists,up 9b6a88b5-69a3-45a1-ae21-0977422f0e62 2024-02-05T06:17:48.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:17:48.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:17:48.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:17:48.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-02-05T06:17:48.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T06:17:49.237 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T06:17:49.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T06:17:49.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T06:17:49.794 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T06:17:49.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:17:50.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T06:17:54.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-02-05T06:17:54.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:17:54.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:17:54.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:17:54.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:17:54.540 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:17:54.540 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:17:54.544 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:17:54.544 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:17:54.544 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:17:54.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:17:54.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:17:54.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:17:54.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:17:54.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:17:54.679 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:17:55.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:17:55.069 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:17:55.069 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:17:55.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:17:55.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:17:55.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:17:55.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-02-05T06:17:55.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-02-05T06:17:55.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-02-05T06:17:55.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:17:55.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:17:55.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-02-05T06:17:55.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-02-05T06:17:55.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672981' 2024-02-05T06:17:55.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:17:55.365 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:17:55.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-02-05T06:17:55.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-02-05T06:17:55.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672981 2-64424509454' 2024-02-05T06:17:55.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:17:55.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-02-05T06:17:55.507 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:17:55.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:17:55.508 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-02-05T06:17:55.508 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:17:55.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-02-05T06:17:55.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-02-05T06:17:55.510 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836507 2024-02-05T06:17:55.510 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:17:55.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-02-05T06:17:55.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:17:56.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:17:56.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:17:57.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-02-05T06:17:57.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:17:57.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:17:57.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-02-05T06:17:57.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:17:57.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-02-05T06:17:57.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:17:57.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-02-05T06:17:57.135 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672981 2024-02-05T06:17:57.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-02-05T06:17:57.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:17:57.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672981 2024-02-05T06:17:57.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:17:57.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-02-05T06:17:57.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:17:57.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:17:57.489 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:17:57.489 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-02-05T06:17:57.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-02-05T06:17:57.490 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509454 2024-02-05T06:17:57.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-02-05T06:17:57.490 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:17:57.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-02-05T06:17:57.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:17:57.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:17:57.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:17:58.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T06:17:58.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:17:58.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:17:58.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:17:58.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:17:58.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:17:58.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:17:58.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:17:58.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:17:58.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:17:58.513 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:17:58.513 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:17:58.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T06:17:58.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:17:58.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:17:58.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-02-05T06:17:58.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T06:17:58.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T06:17:58.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-02-05T06:17:58.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T06:17:58.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-02-05T06:17:59.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T06:17:59.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T06:18:00.413 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'ecpool' created 2024-02-05T06:18:00.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:18:01.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-02-05T06:18:01.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T06:18:01.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:18:01.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:18:01.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:18:01.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:18:01.440 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:18:01.440 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:18:01.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:18:01.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:18:01.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:18:01.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:18:01.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:18:01.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:18:01.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:18:01.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:18:01.578 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:18:01.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:18:01.890 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:18:01.891 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:18:01.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:18:01.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:01.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:18:02.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-02-05T06:18:02.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-02-05T06:18:02.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-02-05T06:18:02.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:02.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:18:02.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-02-05T06:18:02.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-02-05T06:18:02.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672984' 2024-02-05T06:18:02.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:02.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:18:02.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-02-05T06:18:02.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-02-05T06:18:02.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672984 2-64424509456' 2024-02-05T06:18:02.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:02.366 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-02-05T06:18:02.367 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:02.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:18:02.370 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-02-05T06:18:02.370 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:02.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-02-05T06:18:02.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-02-05T06:18:02.371 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836509 2024-02-05T06:18:02.371 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:02.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-02-05T06:18:02.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:02.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-02-05T06:18:02.675 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:02.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:18:02.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-02-05T06:18:02.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:02.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-02-05T06:18:02.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-02-05T06:18:02.680 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672984 2024-02-05T06:18:02.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:18:03.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-02-05T06:18:03.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:03.043 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-02-05T06:18:03.043 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:03.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:18:03.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-02-05T06:18:03.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:03.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-02-05T06:18:03.047 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509456 2024-02-05T06:18:03.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-02-05T06:18:03.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:18:03.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509456 2024-02-05T06:18:03.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:18:04.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:18:04.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:18:04.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-02-05T06:18:04.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:18:04.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:18:04.744 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:18:05.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T06:18:05.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:18:05.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:18:05.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:18:05.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:18:05.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:18:05.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:18:05.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:18:05.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:18:05.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:18:05.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:18:05.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:18:05.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T06:18:05.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T06:18:05.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:18:05.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T06:18:05.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:18:05.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:18:05.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:18:05.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:18:05.879 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:18:05.879 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:18:05.879 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:18:05.879 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:18:05.880 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:18:05.880 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:18:06.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:18:06.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:18:06.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:18:06.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:18:06.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T06:18:06.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T06:18:06.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:18:06.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:18:06.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:18:06.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:18:06.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:18:06.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:18:06.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:18:06.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:18:06.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:18:06.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:18:06.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:18:06.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:18:07.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:18:07.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:18:07.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:18:07.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:18:07.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:18:07.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:18:07.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:18:07.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:18:07.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T06:18:07.423 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:18:07.423 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:18:07.424 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:18:07.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T06:18:07.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:18:07.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:18:07.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T06:18:07.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-02-05T06:18:07.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:18:07.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:18:07.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-02-05T06:18:07.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:18:07.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:18:07.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:18:08.230 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:18:08.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:18:08.800 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:18:08.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:18:08.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:18:08.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:18:09.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-02-05T06:18:09.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T06:18:09.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-02-05T06:18:09.300 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:18:09.300 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-02-05T06:18:09.654 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-02-05T06:18:09.654 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:18:09.654 INFO:tasks.workunit.client.0.smithi148.stderr:0' 2024-02-05T06:18:09.654 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-02-05T06:18:09.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-02-05T06:18:09.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-02-05T06:18:09.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-02-05T06:18:09.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:18:09.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:18:09.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-02-05T06:18:09.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:18:09.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:18:09.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:18:10.121 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:18:10.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:18:10.790 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:18:10.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:18:10.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:18:10.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:18:11.365 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-02-05T06:18:11.365 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T06:18:11.366 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-02-05T06:18:11.366 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-02-05T06:18:11.366 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:18:11.767 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-02-05T06:18:11.767 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:18:11.767 INFO:tasks.workunit.client.0.smithi148.stderr:0' 2024-02-05T06:18:11.767 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-02-05T06:18:11.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-02-05T06:18:11.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-02-05T06:18:11.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-02-05T06:18:11.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:18:11.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:18:11.770 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:18:11.770 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:18:11.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:18:11.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:18:11.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:18:13.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:18:13.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:18:13.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:18:13.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:18:13.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:18:13.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:18:13.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:18:13.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:18:14.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:18:14.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:18:14.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T06:18:14.400 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:18:14.400 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:18:14.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:18:14.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:18:14.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:18:15.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:18:15.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-02-05T06:18:15.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-02-05T06:18:17.832 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-02-05T06:18:18.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-02-05T06:18:18.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-02-05T06:18:20.193 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-02-05T06:18:20.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-02-05T06:18:20.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-02-05T06:18:21.856 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-02-05T06:18:22.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-02-05T06:18:22.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-02-05T06:18:24.176 INFO:tasks.workunit.client.0.smithi148.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-02-05T06:18:24.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-02-05T06:18:24.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:18:24.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-02-05T06:18:24.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:18:24.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:18:24.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:18:24.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:18:24.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:18:24.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:18:24.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:18:24.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:18:24.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:18:24.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:18:24.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:18:24.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:18:24.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:18:24.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:18:24.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:18:24.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:18:24.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:18:24.720 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:18:24.720 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:18:24.720 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:18:24.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:18:24.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:18:24.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:18:24.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:18:24.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:18:24.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:18:24.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:18:24.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:18:24.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:18:24.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:18:24.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:18:24.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:18:24.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:18:24.722 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:18:24.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:18:24.722 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:18:24.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:18:24.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:18:24.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:18:24.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:18:24.756 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:24.780+0000 7f1922d5c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:18:24.759 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:24.784+0000 7f1922d5c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:18:24.761 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:24.784+0000 7f1922d5c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:18:25.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:18:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:18:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:18:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:18:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:18:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:18:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:18:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:18:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:18:25.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:18:25.301 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:25.324+0000 7f1922d5c640 -1 Falling back to public interface 2024-02-05T06:18:25.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:18:26.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:18:26.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:18:26.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:18:26.347 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:18:26.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:18:26.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:18:26.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:18:27.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:18:27.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:18:27.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:18:27.660 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:18:27.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:18:27.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:18:27.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:18:28.356 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:28.380+0000 7f1922d5c640 -1 osd.0 33 log_to_monitors true 2024-02-05T06:18:28.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:18:28.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:18:28.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:18:28.983 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:18:28.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:18:28.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:18:29.508 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 41 up_thru 18 down_at 34 last_clean_interval [5,33) [v2:127.0.0.1:6802/2530118726,v1:127.0.0.1:6803/2530118726] [v2:127.0.0.1:6804/2530118726,v1:127.0.0.1:6805/2530118726] exists,up b626794e-171b-49c5-aadf-54d16e8cc9b1 2024-02-05T06:18:29.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:18:29.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:18:29.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:18:29.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:18:29.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-02-05T06:18:29.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:18:29.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:18:29.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:18:29.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:18:29.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:18:29.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:18:29.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:18:29.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:18:29.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:18:29.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:18:29.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:18:29.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:18:29.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:18:29.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:18:29.510 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:18:29.510 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:18:29.510 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:18:29.510 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:18:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:18:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:18:29.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:18:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:18:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:18:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:18:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:18:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:18:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:18:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:18:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:18:29.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:18:29.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:18:29.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:18:29.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:18:29.513 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:18:29.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:18:29.513 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:18:29.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:18:29.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:18:29.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:18:29.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:18:29.542 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:29.564+0000 7f8f893dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:18:29.646 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:29.668+0000 7f8f893dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:18:29.647 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:29.672+0000 7f8f893dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:18:29.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:18:29.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:18:29.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:18:29.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:18:29.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:18:29.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:18:29.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:18:29.875 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:18:29.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:18:29.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:18:30.181 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:30.204+0000 7f8f893dc640 -1 Falling back to public interface 2024-02-05T06:18:30.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:18:31.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:18:31.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:18:31.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:18:31.190 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:18:31.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:18:31.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:18:31.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:18:32.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:18:32.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:18:32.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:18:32.510 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:18:32.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:18:32.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:18:32.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:18:33.132 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:33.156+0000 7f8f893dc640 -1 osd.1 36 log_to_monitors true 2024-02-05T06:18:33.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:18:33.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:18:33.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:18:33.821 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:18:33.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:18:33.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:18:34.264 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/1928572714,v1:127.0.0.1:6811/1928572714] [v2:127.0.0.1:6812/1928572714,v1:127.0.0.1:6813/1928572714] exists,up efe4b562-1466-4b99-8b08-f5760603f851 2024-02-05T06:18:34.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:18:34.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:18:34.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:18:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:18:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-02-05T06:18:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:18:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:18:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T06:18:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:18:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:18:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:18:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:18:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:18:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:18:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:18:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:18:34.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:18:34.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:18:34.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:18:34.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:18:34.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:18:34.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:18:34.267 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:18:34.267 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:18:34.267 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:18:34.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:18:34.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:18:34.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:18:34.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:18:34.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:18:34.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:18:34.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:18:34.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:18:34.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:18:34.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:18:34.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:18:34.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:18:34.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T06:18:34.268 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.2 2024-02-05T06:18:34.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:18:34.269 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T06:18:34.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T06:18:34.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:18:34.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:18:34.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:18:34.299 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:34.324+0000 7ff9ea839640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:18:34.503 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:34.524+0000 7ff9ea839640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:18:34.505 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:34.528+0000 7ff9ea839640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:18:34.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T06:18:34.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:18:34.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:18:34.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:18:34.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:18:34.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:18:34.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:18:34.728 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:18:34.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:18:34.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:18:35.042 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:35.064+0000 7ff9ea839640 -1 Falling back to public interface 2024-02-05T06:18:35.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:18:36.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:18:36.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:18:36.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:18:36.149 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:18:36.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:18:36.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:18:36.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:18:37.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:18:37.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:18:37.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:18:37.468 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:18:37.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:18:37.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:18:37.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:18:38.112 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:38.135+0000 7ff9ea839640 -1 osd.2 38 log_to_monitors true 2024-02-05T06:18:38.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:18:38.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:18:38.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:18:38.787 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:18:38.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:18:38.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:18:39.291 INFO:tasks.workunit.client.0.smithi148.stderr:osd.2 up in weight 1 up_from 47 up_thru 37 down_at 39 last_clean_interval [15,38) [v2:127.0.0.1:6818/3579067643,v1:127.0.0.1:6819/3579067643] [v2:127.0.0.1:6820/3579067643,v1:127.0.0.1:6821/3579067643] exists,up 9b6a88b5-69a3-45a1-ae21-0977422f0e62 2024-02-05T06:18:39.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:18:39.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:18:39.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:18:39.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-02-05T06:18:39.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T06:18:40.551 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' removed 2024-02-05T06:18:40.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T06:18:40.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T06:18:41.133 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T06:18:41.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:18:42.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T06:18:44.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-02-05T06:18:44.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:18:44.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:18:44.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:18:44.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:18:44.610 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:18:44.610 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:18:44.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:18:44.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:18:44.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:18:44.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:18:44.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:18:44.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:18:44.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:18:44.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:18:44.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:18:45.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:18:45.086 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:18:45.086 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:18:45.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:18:45.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:45.087 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:18:45.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659141 2024-02-05T06:18:45.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659141 2024-02-05T06:18:45.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659141' 2024-02-05T06:18:45.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:45.235 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:18:45.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561028 2024-02-05T06:18:45.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561028 2024-02-05T06:18:45.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659141 1-188978561028' 2024-02-05T06:18:45.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:45.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:18:45.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-02-05T06:18:45.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-02-05T06:18:45.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659141 1-188978561028 2-201863462915' 2024-02-05T06:18:45.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:45.519 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-176093659141 2024-02-05T06:18:45.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:45.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:18:45.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-176093659141 2024-02-05T06:18:45.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:45.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659141 2024-02-05T06:18:45.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 176093659141' 2024-02-05T06:18:45.522 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 176093659141 2024-02-05T06:18:45.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:45.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659140 -lt 176093659141 2024-02-05T06:18:45.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:18:46.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:18:46.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:47.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659141 -lt 176093659141 2024-02-05T06:18:47.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:47.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561028 2024-02-05T06:18:47.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:47.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:18:47.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561028 2024-02-05T06:18:47.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:47.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561028 2024-02-05T06:18:47.144 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 188978561028 2024-02-05T06:18:47.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561028' 2024-02-05T06:18:47.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:18:47.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561028 -lt 188978561028 2024-02-05T06:18:47.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:47.483 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-02-05T06:18:47.483 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:47.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:18:47.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-02-05T06:18:47.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:47.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-02-05T06:18:47.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-02-05T06:18:47.488 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 201863462915 2024-02-05T06:18:47.488 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:18:47.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462915 -lt 201863462915 2024-02-05T06:18:47.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:18:47.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:18:47.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:18:48.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T06:18:48.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:18:48.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:18:48.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:18:48.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:18:48.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:18:48.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:18:48.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:18:48.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T06:18:48.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:18:48.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:18:48.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:18:48.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T06:18:48.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:18:48.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:18:48.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-02-05T06:18:48.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T06:18:48.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-02-05T06:18:48.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-02-05T06:18:48.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:18:49.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-02-05T06:18:49.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-02-05T06:18:49.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T06:18:49.204 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T06:18:49.204 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:18:49.204 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:18:49.204 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:18:49.204 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:18:49.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T06:18:00.315308+0000 2024-02-05T06:18:49.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T06:18:49.823 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-02-05T06:18:49.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T06:18:00.315308+0000 2024-02-05T06:18:49.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T06:18:49.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:18:00.315308+0000 2024-02-05T06:18:49.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:18:49.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:18:49.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:18:49.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:18:49.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:18:49.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:18:49.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:18:49.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:18:50.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:18:00.315308+0000 '>' 2024-02-05T06:18:00.315308+0000 2024-02-05T06:18:50.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:18:51.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:18:51.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:18:51.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:18:51.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:18:51.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:18:51.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:18:51.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:18:51.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:18:00.315308+0000 '>' 2024-02-05T06:18:00.315308+0000 2024-02-05T06:18:51.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:18:52.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:18:52.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:18:52.451 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:18:52.451 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:18:52.451 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:18:52.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:18:52.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:18:52.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:18:00.315308+0000 '>' 2024-02-05T06:18:00.315308+0000 2024-02-05T06:18:52.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:18:53.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:18:53.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:18:53.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:18:53.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:18:53.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:18:53.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:18:53.756 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:18:54.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:18:00.315308+0000 '>' 2024-02-05T06:18:00.315308+0000 2024-02-05T06:18:54.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:18:55.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:18:55.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:18:55.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:18:55.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:18:55.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:18:55.062 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:18:55.062 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:18:55.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:18:50.035158+0000 '>' 2024-02-05T06:18:00.315308+0000 2024-02-05T06:18:55.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:18:55.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-02-05T06:18:55.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-02-05T06:18:55.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-02-05T06:18:55.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-02-05T06:18:55.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-02-05T06:18:55.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-02-05T06:18:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-02-05T06:18:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-02-05T06:18:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-02-05T06:18:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:18:55.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:18:55.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:18:55.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:18:55.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:18:55.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:18:55.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:18:55.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:18:55.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:18:55.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:18:55.634 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:18:55.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:18:55.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:18:55.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:18:55.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:18:55.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:18:55.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:18:55.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:18:55.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:18:55.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:18:55.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:18:55.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:18:55.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:18:55.659 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:18:55.659 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:18:55.659 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:18:55.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:18:55.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:18:55.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:18:55.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:18:55.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:18:55.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:18:55.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:18:55.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:18:55.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:18:55.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:18:55.662 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:18:55.662 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:18:55.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:18:55.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:18:55.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:18:55.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:18:55.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:18:55.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:18:55.665 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:18:55.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:18:55.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:18:55.667 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:18:55.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:18:55.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:18:55.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:18:55.669 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:18:55.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:18:55.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:18:55.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:18:55.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:18:55.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:18:55.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:18:55.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:18:55.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:18:55.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:18:55.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:18:55.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:18:55.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:18:55.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:18:55.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T06:18:55.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:18:55.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:18:55.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:18:55.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:18:55.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-02-05T06:18:55.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-02-05T06:18:55.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-02-05T06:18:55.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T06:18:55.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-02-05T06:18:55.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-02-05T06:18:55.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-02-05T06:18:55.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:18:55.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:18:55.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:18:55.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:18:55.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:18:55.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T06:18:56.129 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:18:56.130 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:18:56.130 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:18:56.130 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:18:56.130 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:18:56.130 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:18:56.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:18:56.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T06:18:56.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:18:56.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:18:56.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:18:56.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:18:56.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:18:56.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:18:56.350 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:18:56.350 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:18:56.350 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:18:56.350 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:18:56.350 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:18:56.351 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:18:56.351 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:18:56.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:18:56.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T06:18:56.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:18:56.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:18:56.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:18:56.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:18:56.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:18:56.469 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:18:56.469 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:18:56.469 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:18:56.469 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:18:56.469 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:18:56.469 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:18:56.469 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:18:56.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:18:56.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T06:18:56.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-02-05T06:18:56.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:18:56.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:18:56.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:18:56.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:18:56.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:18:56.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:18:58.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:18:58.275 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:18:58.275 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:18:58.275 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:18:58.275 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:18:58.276 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:18:58.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:18:58.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:18:58.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:18:58.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-02-05T06:18:58.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:18:58.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-02-05T06:18:58.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:18:58.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:18:58.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:18:58.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:18:58.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:18:58.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:18:58.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:18:58.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:18:58.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:18:58.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:18:58.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:18:58.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:18:58.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:18:58.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:18:58.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:18:58.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:18:58.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:18:58.319 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:18:58.319 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:18:58.319 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:18:58.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:18:58.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:18:58.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:18:58.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:18:58.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:18:58.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:18:58.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:18:58.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:18:58.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:18:58.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:18:58.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:18:58.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:18:58.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:18:58.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:18:58.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:18:58.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f2ad6f9d-c39e-4e44-b256-c3a6cd856e81 2024-02-05T06:18:58.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f2ad6f9d-c39e-4e44-b256-c3a6cd856e81' 2024-02-05T06:18:58.321 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 f2ad6f9d-c39e-4e44-b256-c3a6cd856e81 2024-02-05T06:18:58.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:18:58.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSfcBl3cSNFRAA980CKzY8HRiwol24lOZA6g== 2024-02-05T06:18:58.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSfcBl3cSNFRAA980CKzY8HRiwol24lOZA6g=="}' 2024-02-05T06:18:58.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f2ad6f9d-c39e-4e44-b256-c3a6cd856e81 -i td/osd-scrub-repair/0/new.json 2024-02-05T06:18:58.645 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:18:58.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:18:58.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSfcBl3cSNFRAA980CKzY8HRiwol24lOZA6g== --osd-uuid f2ad6f9d-c39e-4e44-b256-c3a6cd856e81 2024-02-05T06:18:58.701 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:58.723+0000 7f42718a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:18:58.729 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:58.751+0000 7f42718a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:18:58.731 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:58.751+0000 7f42718a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:18:58.731 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:58.751+0000 7f42718a4640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:18:58.731 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:58.751+0000 7f42718a4640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:18:58.731 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:58.751+0000 7f42718a4640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:18:58.829 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:18:58.851+0000 7f42718a4640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:19:07.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:19:07.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:19:07.303 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:19:07.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:19:07.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:19:07.693 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:19:07.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:19:07.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:19:07.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:19:07.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:19:07.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:19:07.725 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:07.743+0000 7ffad65f0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:19:07.727 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:07.747+0000 7ffad65f0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:19:07.728 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:07.747+0000 7ffad65f0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:19:08.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:19:08.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:19:08.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:19:08.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:19:08.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:19:08.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:08.002 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:19:08.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:19:08.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:08.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:08.311 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:08.331+0000 7ffad65f0640 -1 Falling back to public interface 2024-02-05T06:19:08.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:09.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:09.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:09.314 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:19:09.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:19:09.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:09.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:09.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:10.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:10.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:10.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:19:10.632 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:19:10.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:10.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:10.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:11.211 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:11.231+0000 7ffad65f0640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:19:11.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:11.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:11.953 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:19:11.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:19:11.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:11.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:12.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:13.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:13.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:13.272 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:19:13.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:19:13.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:13.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:13.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:14.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:14.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:14.730 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:19:14.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:19:14.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:14.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:15.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:16.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:16.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:16.044 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:19:16.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:19:16.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:16.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:16.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:17.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:17.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:17.667 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:19:17.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:19:17.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:17.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:17.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:18.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:18.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:18.983 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:19:18.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:19:18.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:18.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:19.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:20.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:20.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:20.298 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:19:20.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:19:20.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:20.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:20.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:21.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:21.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:21.644 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:19:21.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:19:21.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:21.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:21.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:22.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:22.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:22.953 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:19:22.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:19:22.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:22.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:23.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:24.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:24.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:24.268 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:19:24.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:19:24.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:24.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:24.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:25.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:25.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:25.714 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:19:25.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:19:25.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:25.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:26.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:27.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:27.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:27.207 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:19:27.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:19:27.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:27.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:27.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:28.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:28.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:28.746 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:19:28.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:19:28.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:28.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:29.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:30.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:30.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:30.088 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:19:30.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:19:30.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:30.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:30.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:31.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:31.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:31.439 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:19:31.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:19:31.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:31.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:31.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:32.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:32.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:32.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:19:32.811 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:19:32.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:32.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:33.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:34.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:34.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:34.710 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:19:34.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:19:34.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:34.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:35.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:36.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:36.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:36.332 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:19:36.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:19:36.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:36.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:36.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:37.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:37.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:37.787 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T06:19:37.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T06:19:37.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:37.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:38.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:39.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:39.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:39.431 INFO:tasks.workunit.client.0.smithi148.stdout:22 2024-02-05T06:19:39.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-02-05T06:19:39.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:39.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:40.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:41.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:41.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:41.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-02-05T06:19:41.070 INFO:tasks.workunit.client.0.smithi148.stdout:23 2024-02-05T06:19:41.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:41.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:41.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:42.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:42.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:42.743 INFO:tasks.workunit.client.0.smithi148.stdout:24 2024-02-05T06:19:42.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-02-05T06:19:42.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:42.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:43.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:44.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:44.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:44.309 INFO:tasks.workunit.client.0.smithi148.stdout:25 2024-02-05T06:19:44.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-02-05T06:19:44.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:44.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:44.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:45.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:45.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:45.653 INFO:tasks.workunit.client.0.smithi148.stdout:26 2024-02-05T06:19:45.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-02-05T06:19:45.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:45.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:19:45.955 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3450822108,v1:127.0.0.1:6803/3450822108] [v2:127.0.0.1:6804/3450822108,v1:127.0.0.1:6805/3450822108] exists,up f2ad6f9d-c39e-4e44-b256-c3a6cd856e81 2024-02-05T06:19:45.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:19:45.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:19:45.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:19:45.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:19:45.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-02-05T06:19:45.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:19:45.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:19:45.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:19:45.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:19:45.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:19:45.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:19:45.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:19:45.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:19:45.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:19:45.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:19:45.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:19:45.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:19:45.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:19:45.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:19:45.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:19:45.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:19:45.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:19:45.959 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:19:45.959 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:19:45.960 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:19:45.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:19:45.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:19:45.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:19:45.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:19:45.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:19:45.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:19:45.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:19:45.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:19:45.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:19:45.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:19:45.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:19:45.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:19:45.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:19:45.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:19:45.962 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:19:45.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5708d18f-ce04-4c1e-adc0-4addb49df7e9 2024-02-05T06:19:45.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5708d18f-ce04-4c1e-adc0-4addb49df7e9' 2024-02-05T06:19:45.963 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 5708d18f-ce04-4c1e-adc0-4addb49df7e9 2024-02-05T06:19:45.963 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:19:45.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACfsBl6gUOABAAV8Udo6MEbUWZGqSM0134aA== 2024-02-05T06:19:45.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACfsBl6gUOABAAV8Udo6MEbUWZGqSM0134aA=="}' 2024-02-05T06:19:45.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5708d18f-ce04-4c1e-adc0-4addb49df7e9 -i td/osd-scrub-repair/1/new.json 2024-02-05T06:19:46.379 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:19:46.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T06:19:46.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACfsBl6gUOABAAV8Udo6MEbUWZGqSM0134aA== --osd-uuid 5708d18f-ce04-4c1e-adc0-4addb49df7e9 2024-02-05T06:19:46.438 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:46.458+0000 7f27f96ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:19:46.515 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:46.534+0000 7f27f96ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:19:46.517 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:46.534+0000 7f27f96ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:19:46.517 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:46.534+0000 7f27f96ca640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:19:46.517 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:46.534+0000 7f27f96ca640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:19:46.518 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:46.534+0000 7f27f96ca640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:19:46.604 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:46.626+0000 7f27f96ca640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T06:19:55.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T06:19:55.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:19:55.350 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:19:55.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:19:55.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:19:55.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:19:55.768 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:19:55.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:19:55.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:19:55.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:19:55.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:19:55.802 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:55.822+0000 7fe6e5fa0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:19:55.847 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:55.870+0000 7fe6e5fa0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:19:55.849 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:55.870+0000 7fe6e5fa0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:19:56.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:19:56.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:19:56.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:19:56.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:19:56.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:19:56.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:56.060 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:19:56.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:19:56.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:19:56.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:56.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:56.413 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:56.434+0000 7fe6e5fa0640 -1 Falling back to public interface 2024-02-05T06:19:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:57.415 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:19:57.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:19:57.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:57.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:19:57.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:58.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:19:58.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:19:58.717 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:19:58.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:19:58.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:19:58.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:19:59.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:19:59.361 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:19:59.382+0000 7fe6e5fa0640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:20:00.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:00.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:00.117 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:20:00.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:20:00.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:00.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:00.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:01.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:01.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:01.585 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:20:01.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:20:01.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:01.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:02.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:03.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:03.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:03.038 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:20:03.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:20:03.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:03.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:03.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:04.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:04.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:04.460 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:20:04.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:20:04.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:04.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:04.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:05.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:05.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:05.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:20:05.781 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:20:05.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:05.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:06.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:07.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:07.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:07.101 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:20:07.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:20:07.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:07.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:07.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:08.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:08.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:08.534 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:20:08.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:20:08.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:08.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:08.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:09.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:09.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:09.963 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:20:09.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:20:09.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:09.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:10.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:11.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:11.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:11.283 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:20:11.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:20:11.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:11.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:11.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:12.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:12.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:12.617 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:20:12.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:20:12.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:12.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:12.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:13.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:13.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:13.956 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:20:13.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:20:13.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:13.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:14.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:15.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:15.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:15.273 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:20:15.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:20:15.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:15.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:15.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:16.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:16.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:16.778 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:20:16.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:20:16.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:16.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:17.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:18.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:18.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:18.130 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:20:18.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:20:18.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:18.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:18.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:19.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:19.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:19.644 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:20:19.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:20:19.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:19.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:20.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:21.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:21.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:21.271 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:20:21.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:20:21.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:21.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:21.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:22.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:22.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:22.709 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:20:22.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:20:22.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:22.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:23.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:24.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:24.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:24.634 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:20:24.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:20:24.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:24.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:25.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:26.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:26.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:26.585 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T06:20:26.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T06:20:26.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:26.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:27.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:28.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:28.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:28.701 INFO:tasks.workunit.client.0.smithi148.stdout:22 2024-02-05T06:20:28.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-02-05T06:20:28.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:28.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:29.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:30.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:30.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:30.512 INFO:tasks.workunit.client.0.smithi148.stdout:23 2024-02-05T06:20:30.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-02-05T06:20:30.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:30.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:31.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:32.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:32.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:32.244 INFO:tasks.workunit.client.0.smithi148.stdout:24 2024-02-05T06:20:32.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-02-05T06:20:32.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:32.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:32.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:33.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:33.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:33.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-02-05T06:20:33.739 INFO:tasks.workunit.client.0.smithi148.stdout:25 2024-02-05T06:20:33.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:33.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:20:34.049 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1522210825,v1:127.0.0.1:6811/1522210825] [v2:127.0.0.1:6812/1522210825,v1:127.0.0.1:6813/1522210825] exists,up 5708d18f-ce04-4c1e-adc0-4addb49df7e9 2024-02-05T06:20:34.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:20:34.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:20:34.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:20:34.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:20:34.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-02-05T06:20:34.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:20:34.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:20:34.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T06:20:34.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:20:34.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:20:34.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:20:34.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:20:34.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:20:34.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:20:34.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:20:34.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:20:34.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:20:34.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:20:34.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:20:34.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:20:34.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:20:34.054 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:20:34.054 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:20:34.054 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:20:34.054 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:20:34.054 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:20:34.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:20:34.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:20:34.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:20:34.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:20:34.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:20:34.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:20:34.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:20:34.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:20:34.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:20:34.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:20:34.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:20:34.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:20:34.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:20:34.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:20:34.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5319623f-701c-4fce-9620-3b5891f3ad37 2024-02-05T06:20:34.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5319623f-701c-4fce-9620-3b5891f3ad37' 2024-02-05T06:20:34.057 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 5319623f-701c-4fce-9620-3b5891f3ad37 2024-02-05T06:20:34.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:20:34.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAyfsBldn3LBRAAgPIYytvNLzpoeR2fnO+1kA== 2024-02-05T06:20:34.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAyfsBldn3LBRAAgPIYytvNLzpoeR2fnO+1kA=="}' 2024-02-05T06:20:34.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5319623f-701c-4fce-9620-3b5891f3ad37 -i td/osd-scrub-repair/2/new.json 2024-02-05T06:20:34.614 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:20:34.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T06:20:34.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAyfsBldn3LBRAAgPIYytvNLzpoeR2fnO+1kA== --osd-uuid 5319623f-701c-4fce-9620-3b5891f3ad37 2024-02-05T06:20:34.674 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:20:34.693+0000 7f2a91a5b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:20:34.758 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:20:34.777+0000 7f2a91a5b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:20:34.759 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:20:34.777+0000 7f2a91a5b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:20:34.760 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:20:34.777+0000 7f2a91a5b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:20:34.760 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:20:34.777+0000 7f2a91a5b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:20:34.760 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:20:34.777+0000 7f2a91a5b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:20:34.857 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:20:34.877+0000 7f2a91a5b640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T06:20:43.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T06:20:43.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:20:43.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T06:20:43.166 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T06:20:43.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:20:43.540 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T06:20:43.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T06:20:43.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:20:43.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:20:43.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:20:43.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:20:43.573 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:20:43.589+0000 7f993dc68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:20:43.624 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:20:43.641+0000 7f993dc68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:20:43.627 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:20:43.645+0000 7f993dc68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:20:43.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T06:20:43.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:20:43.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:20:43.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:20:43.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:20:43.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:43.848 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:20:43.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:20:43.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:43.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:20:44.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:44.227 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:20:44.245+0000 7f993dc68640 -1 Falling back to public interface 2024-02-05T06:20:45.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:45.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:45.143 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:20:45.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:20:45.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:45.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:20:45.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:46.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:46.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:46.456 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:20:46.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:20:46.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:46.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:20:46.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:47.242 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:20:47.261+0000 7f993dc68640 -1 osd.2 0 log_to_monitors true 2024-02-05T06:20:47.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:47.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:47.774 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:20:47.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:20:47.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:47.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:20:48.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:49.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:49.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:49.371 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:20:49.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:20:49.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:49.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:20:49.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:50.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:50.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:50.788 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:20:50.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:20:50.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:50.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:20:51.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:52.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:52.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:52.120 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:20:52.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:20:52.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:52.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:20:52.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:53.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:53.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:53.525 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:20:53.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:20:53.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:53.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:20:53.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:54.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:54.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:54.877 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:20:54.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:20:54.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:54.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:20:55.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:56.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:56.287 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:20:56.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:20:56.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:56.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:20:56.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:57.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:57.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:57.668 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:20:57.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:20:57.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:57.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:20:58.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:20:59.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:20:59.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:20:59.040 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:20:59.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:20:59.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:20:59.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:20:59.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:00.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:00.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:00.468 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:21:00.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:21:00.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:00.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:00.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:01.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:01.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:01.790 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:21:01.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:21:01.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:01.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:02.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:03.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:03.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:03.107 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:21:03.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:21:03.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:03.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:03.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:04.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:04.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:04.556 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:21:04.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:21:04.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:04.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:04.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:05.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:05.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:05.860 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:21:05.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:21:05.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:05.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:06.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:07.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:07.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:07.264 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:21:07.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:21:07.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:07.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:07.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:08.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:08.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:08.752 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:21:08.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:21:08.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:08.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:09.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:10.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:10.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:10.792 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:21:10.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:21:10.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:10.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:11.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:12.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:12.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:12.499 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:21:12.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:21:12.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:12.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:13.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:14.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:14.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:14.237 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T06:21:14.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T06:21:14.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:14.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:14.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:15.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:15.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:15.705 INFO:tasks.workunit.client.0.smithi148.stdout:22 2024-02-05T06:21:15.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-02-05T06:21:15.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:15.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:16.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:17.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:17.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:17.336 INFO:tasks.workunit.client.0.smithi148.stdout:23 2024-02-05T06:21:17.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-02-05T06:21:17.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:17.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:17.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:18.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:18.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:18.647 INFO:tasks.workunit.client.0.smithi148.stdout:24 2024-02-05T06:21:18.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-02-05T06:21:18.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:18.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:18.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:19.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:19.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:19.966 INFO:tasks.workunit.client.0.smithi148.stdout:25 2024-02-05T06:21:19.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-02-05T06:21:19.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:19.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:20.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:21.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:21.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:21.465 INFO:tasks.workunit.client.0.smithi148.stdout:26 2024-02-05T06:21:21.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-02-05T06:21:21.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:21.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:21:21.871 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1992007729,v1:127.0.0.1:6819/1992007729] [v2:127.0.0.1:6820/1992007729,v1:127.0.0.1:6821/1992007729] exists,up 5319623f-701c-4fce-9620-3b5891f3ad37 2024-02-05T06:21:21.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:21:21.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:21:21.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:21:21.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-02-05T06:21:21.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T06:21:22.172 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T06:21:22.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T06:21:22.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T06:21:22.667 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T06:21:22.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:21:23.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T06:21:26.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-02-05T06:21:26.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:21:26.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:21:26.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:21:26.424 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:21:26.425 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:21:26.425 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:21:26.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:21:26.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:21:26.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:21:26.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:21:26.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:21:26.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:21:26.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:21:26.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:21:26.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:21:26.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:21:26.926 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:21:26.926 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:21:26.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:21:26.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:21:26.926 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:21:27.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-02-05T06:21:27.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-02-05T06:21:27.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-02-05T06:21:27.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:21:27.075 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:21:27.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-02-05T06:21:27.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-02-05T06:21:27.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672978' 2024-02-05T06:21:27.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:21:27.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:21:27.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-02-05T06:21:27.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-02-05T06:21:27.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672978 2-64424509447' 2024-02-05T06:21:27.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:21:27.362 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-02-05T06:21:27.362 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:21:27.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:21:27.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-02-05T06:21:27.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:21:27.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-02-05T06:21:27.365 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836507 2024-02-05T06:21:27.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-02-05T06:21:27.366 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:21:27.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-02-05T06:21:27.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:21:28.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:21:28.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:21:28.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-02-05T06:21:28.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:21:28.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-02-05T06:21:28.984 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:21:28.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:21:28.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-02-05T06:21:28.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:21:28.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-02-05T06:21:28.988 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672978 2024-02-05T06:21:28.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-02-05T06:21:28.989 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:21:29.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-02-05T06:21:29.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:21:29.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-02-05T06:21:29.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:21:29.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:21:29.284 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-02-05T06:21:29.284 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:21:29.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-02-05T06:21:29.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-02-05T06:21:29.285 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509447 2024-02-05T06:21:29.286 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:21:29.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-02-05T06:21:29.723 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:21:29.723 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:21:29.723 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:21:30.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T06:21:30.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:21:30.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:21:30.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:21:30.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:21:30.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:21:30.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:21:30.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:21:30.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:21:30.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:21:30.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:21:30.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:21:30.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T06:21:30.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:21:30.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:21:30.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-02-05T06:21:30.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T06:21:30.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T06:21:30.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-02-05T06:21:30.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T06:21:30.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-02-05T06:21:31.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T06:21:31.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T06:21:32.279 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'ecpool' created 2024-02-05T06:21:32.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:21:33.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-02-05T06:21:33.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-02-05T06:21:34.145 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 2 allow_ec_overwrites to true 2024-02-05T06:21:34.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T06:21:34.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:21:34.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:21:34.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:21:34.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:21:34.168 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:21:34.169 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:21:34.169 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:21:34.169 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:21:34.169 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:21:34.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:21:34.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:21:34.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:21:34.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:21:34.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:21:34.285 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:21:34.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:21:34.733 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:21:34.733 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:21:34.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:21:34.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:21:34.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:21:34.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-02-05T06:21:34.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-02-05T06:21:34.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-02-05T06:21:34.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:21:34.884 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:21:35.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-02-05T06:21:35.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-02-05T06:21:35.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672981' 2024-02-05T06:21:35.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:21:35.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:21:35.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-02-05T06:21:35.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-02-05T06:21:35.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672981 2-64424509449' 2024-02-05T06:21:35.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:21:35.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:21:35.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-02-05T06:21:35.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:21:35.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-02-05T06:21:35.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:21:35.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-02-05T06:21:35.206 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836509 2024-02-05T06:21:35.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-02-05T06:21:35.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:21:35.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-02-05T06:21:35.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:21:36.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:21:36.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:21:36.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-02-05T06:21:36.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:21:37.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:21:37.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:21:38.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836509 2024-02-05T06:21:38.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:21:38.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:21:38.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-02-05T06:21:38.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:21:38.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-02-05T06:21:38.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:21:38.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-02-05T06:21:38.114 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672981 2024-02-05T06:21:38.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-02-05T06:21:38.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:21:38.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-02-05T06:21:38.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:21:38.413 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-02-05T06:21:38.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:21:38.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:21:38.417 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:21:38.417 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-02-05T06:21:38.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-02-05T06:21:38.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-02-05T06:21:38.419 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509449 2024-02-05T06:21:38.419 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:21:38.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-02-05T06:21:38.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:21:38.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:21:38.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:21:39.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T06:21:39.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:21:39.215 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:21:39.216 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:21:39.216 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:21:39.216 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:21:39.216 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:21:39.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:21:39.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T06:21:39.519 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:21:39.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:21:39.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:21:39.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T06:21:39.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:21:39.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:21:39.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T06:21:39.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-02-05T06:21:39.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:21:39.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:21:39.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-02-05T06:21:39.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:21:39.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:21:39.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:21:40.348 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:21:40.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:21:40.850 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:21:40.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:21:40.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:21:40.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:21:41.359 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-02-05T06:21:41.359 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T06:21:41.359 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-02-05T06:21:41.360 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-02-05T06:21:41.360 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:21:41.735 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-02-05T06:21:41.735 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:21:41.735 INFO:tasks.workunit.client.0.smithi148.stderr:0' 2024-02-05T06:21:41.735 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-02-05T06:21:41.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-02-05T06:21:41.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-02-05T06:21:41.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-02-05T06:21:41.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:21:41.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:21:41.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-02-05T06:21:41.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:21:41.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:21:41.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:21:42.197 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:21:42.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:21:42.724 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:21:42.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:21:42.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:21:42.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:21:43.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-02-05T06:21:43.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T06:21:43.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-02-05T06:21:43.189 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-02-05T06:21:43.189 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:21:43.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-02-05T06:21:43.522 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:21:43.522 INFO:tasks.workunit.client.0.smithi148.stderr:0' 2024-02-05T06:21:43.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-02-05T06:21:43.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-02-05T06:21:43.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-02-05T06:21:43.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-02-05T06:21:43.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:21:43.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:21:43.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:21:43.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:21:43.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:21:43.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:21:43.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:21:44.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:21:44.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:21:44.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:21:44.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:21:44.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:21:44.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:21:44.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:21:44.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:21:46.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:21:46.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:21:46.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T06:21:46.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:21:46.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:21:46.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:21:46.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:21:46.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:21:47.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:21:47.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-02-05T06:21:47.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-02-05T06:21:49.214 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-02-05T06:21:49.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-02-05T06:21:49.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-02-05T06:21:51.560 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-02-05T06:21:52.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-02-05T06:21:52.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-02-05T06:21:53.298 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-02-05T06:21:53.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-02-05T06:21:53.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-02-05T06:21:55.702 INFO:tasks.workunit.client.0.smithi148.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-02-05T06:21:56.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-02-05T06:21:56.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:21:56.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-02-05T06:21:56.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:21:56.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:21:56.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:21:56.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:21:56.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:21:56.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:21:56.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:21:56.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:21:56.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:21:56.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:21:56.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:21:56.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:21:56.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:21:56.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:21:56.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:21:56.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:21:56.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:21:56.244 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:21:56.245 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:21:56.245 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:21:56.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:21:56.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:21:56.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:21:56.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:21:56.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:21:56.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:21:56.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:21:56.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:21:56.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:21:56.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:21:56.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:21:56.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:21:56.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:21:56.247 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:21:56.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:21:56.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:21:56.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:21:56.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:21:56.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:21:56.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:21:56.283 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:21:56.295+0000 7f14ad8d1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:21:56.285 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:21:56.299+0000 7f14ad8d1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:21:56.287 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:21:56.299+0000 7f14ad8d1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:21:56.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:21:56.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:21:56.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:21:56.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:21:56.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:21:56.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:56.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:21:56.555 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:21:56.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:56.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:21:56.823 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:21:56.835+0000 7f14ad8d1640 -1 Falling back to public interface 2024-02-05T06:21:56.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:57.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:57.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:57.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:21:57.853 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:21:57.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:57.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:21:58.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:21:59.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:21:59.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:21:59.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:21:59.163 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:21:59.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:21:59.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:21:59.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:00.072 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:00.083+0000 7f14ad8d1640 -1 osd.0 33 log_to_monitors true 2024-02-05T06:22:00.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:00.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:00.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:22:00.482 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:22:00.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:00.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:00.967 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 41 up_thru 16 down_at 34 last_clean_interval [5,33) [v2:127.0.0.1:6802/3699039122,v1:127.0.0.1:6803/3699039122] [v2:127.0.0.1:6804/3699039122,v1:127.0.0.1:6805/3699039122] exists,up f2ad6f9d-c39e-4e44-b256-c3a6cd856e81 2024-02-05T06:22:00.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:22:00.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:22:00.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:22:00.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:22:00.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-02-05T06:22:00.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:22:00.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:22:00.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:22:00.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:22:00.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:22:00.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:22:00.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:22:00.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:22:00.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:22:00.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:22:00.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:22:00.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:22:00.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:22:00.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:22:00.969 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:22:00.969 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:22:00.969 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:22:00.971 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:22:00.971 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:22:00.971 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:22:00.971 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:22:00.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:22:00.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:22:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:22:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:22:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:22:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:22:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:22:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:22:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:22:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:22:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:22:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:22:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.1 2024-02-05T06:22:00.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:22:00.973 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:22:00.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:22:00.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:22:00.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:22:00.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:22:00.998 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:01.011+0000 7f39231d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:22:01.095 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:01.107+0000 7f39231d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:22:01.096 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:01.111+0000 7f39231d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:22:01.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:22:01.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:22:01.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:22:01.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:22:01.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:22:01.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:01.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:22:01.286 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:22:01.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:01.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:22:01.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:01.666 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:01.679+0000 7f39231d4640 -1 Falling back to public interface 2024-02-05T06:22:02.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:02.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:02.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:22:02.612 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:22:02.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:02.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:22:02.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:03.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:03.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:03.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:22:03.918 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:22:03.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:03.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:22:04.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:04.459 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:04.471+0000 7f39231d4640 -1 osd.1 36 log_to_monitors true 2024-02-05T06:22:05.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:05.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:05.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:22:05.227 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:22:05.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:05.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:22:05.653 INFO:tasks.workunit.client.0.smithi148.stderr:osd.1 up in weight 1 up_from 44 up_thru 34 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/2986852952,v1:127.0.0.1:6811/2986852952] [v2:127.0.0.1:6812/2986852952,v1:127.0.0.1:6813/2986852952] exists,up 5708d18f-ce04-4c1e-adc0-4addb49df7e9 2024-02-05T06:22:05.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:22:05.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:22:05.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:22:05.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T06:22:05.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-02-05T06:22:05.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:22:05.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:22:05.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T06:22:05.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:22:05.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:22:05.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:22:05.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:22:05.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:22:05.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:22:05.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:22:05.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:22:05.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:22:05.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:22:05.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:22:05.655 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:22:05.655 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:22:05.655 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:22:05.656 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:22:05.656 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:22:05.656 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:22:05.656 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:22:05.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:22:05.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:22:05.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:22:05.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:22:05.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:22:05.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:22:05.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:22:05.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:22:05.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:22:05.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:22:05.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:22:05.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T06:22:05.658 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.2 2024-02-05T06:22:05.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:22:05.659 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T06:22:05.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T06:22:05.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:22:05.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:22:05.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:22:05.688 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:05.699+0000 7f5fc65c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:22:05.691 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:05.703+0000 7f5fc65c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:22:05.692 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:05.707+0000 7f5fc65c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:22:06.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T06:22:06.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:22:06.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:22:06.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:22:06.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:22:06.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:06.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:22:06.035 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:22:06.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:06.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:22:06.243 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:06.255+0000 7f5fc65c8640 -1 Falling back to public interface 2024-02-05T06:22:06.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:07.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:07.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:07.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:22:07.484 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:22:07.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:07.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:22:07.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:08.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:08.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:08.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:22:08.804 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:22:08.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:08.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:22:09.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:09.321 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:09.335+0000 7f5fc65c8640 -1 osd.2 38 log_to_monitors true 2024-02-05T06:22:10.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:10.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:10.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:22:10.131 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:22:10.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:10.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:22:10.538 INFO:tasks.workunit.client.0.smithi148.stderr:osd.2 up in weight 1 up_from 48 up_thru 48 down_at 39 last_clean_interval [15,38) [v2:127.0.0.1:6818/3777242544,v1:127.0.0.1:6819/3777242544] [v2:127.0.0.1:6820/3777242544,v1:127.0.0.1:6821/3777242544] exists,up 5319623f-701c-4fce-9620-3b5891f3ad37 2024-02-05T06:22:10.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:22:10.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:22:10.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:22:10.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-02-05T06:22:10.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T06:22:11.631 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' removed 2024-02-05T06:22:11.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T06:22:11.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T06:22:12.496 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T06:22:12.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:22:13.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T06:22:15.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-02-05T06:22:15.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:22:15.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:22:15.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:22:15.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:22:15.961 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:22:15.961 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:22:15.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:22:15.962 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:22:15.962 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:22:16.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:22:16.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:22:16.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:22:16.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:22:16.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:22:16.090 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:22:16.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:22:16.437 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:22:16.437 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:22:16.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:22:16.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:22:16.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:22:16.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659141 2024-02-05T06:22:16.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659141 2024-02-05T06:22:16.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659141' 2024-02-05T06:22:16.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:22:16.627 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:22:16.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561028 2024-02-05T06:22:16.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561028 2024-02-05T06:22:16.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659141 1-188978561028' 2024-02-05T06:22:16.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:22:16.766 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:22:16.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430211 2024-02-05T06:22:16.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430211 2024-02-05T06:22:16.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659141 1-188978561028 2-206158430211' 2024-02-05T06:22:16.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:22:16.948 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-176093659141 2024-02-05T06:22:16.948 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:22:16.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:22:16.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-176093659141 2024-02-05T06:22:16.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:22:16.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659141 2024-02-05T06:22:16.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 176093659141' 2024-02-05T06:22:16.955 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 176093659141 2024-02-05T06:22:16.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:22:17.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659141 -lt 176093659141 2024-02-05T06:22:17.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:22:17.238 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:22:17.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561028 2024-02-05T06:22:17.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:22:17.241 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:22:17.241 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561028 2024-02-05T06:22:17.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561028 2024-02-05T06:22:17.243 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 188978561028 2024-02-05T06:22:17.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561028' 2024-02-05T06:22:17.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:22:17.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561028 -lt 188978561028 2024-02-05T06:22:17.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:22:17.627 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-206158430211 2024-02-05T06:22:17.628 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:22:17.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:22:17.630 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-206158430211 2024-02-05T06:22:17.630 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:22:17.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430211 2024-02-05T06:22:17.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 206158430211' 2024-02-05T06:22:17.632 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 206158430211 2024-02-05T06:22:17.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:22:17.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430210 -lt 206158430211 2024-02-05T06:22:17.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:22:18.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:22:18.934 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:22:19.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430211 -lt 206158430211 2024-02-05T06:22:19.238 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:22:19.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:22:19.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:22:19.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T06:22:19.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:22:19.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:22:19.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:22:19.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:22:19.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:22:19.653 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:22:19.653 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:22:19.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T06:22:19.941 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:22:19.942 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:22:19.942 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:22:20.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T06:22:20.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:22:20.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:22:20.413 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-02-05T06:22:20.413 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T06:22:20.413 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-02-05T06:22:20.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-02-05T06:22:20.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:22:20.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-02-05T06:22:20.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-02-05T06:22:20.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T06:22:20.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T06:22:20.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:22:20.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:20.725 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:22:20.725 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:21.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T06:21:32.104844+0000 2024-02-05T06:22:21.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T06:22:21.296 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-02-05T06:22:21.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T06:21:32.104844+0000 2024-02-05T06:22:21.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T06:22:21.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:21:32.104844+0000 2024-02-05T06:22:21.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:22:21.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:22:21.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:22:21.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:22:21.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:22:21.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:21.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:21.314 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:22:21.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:21:32.104844+0000 '>' 2024-02-05T06:21:32.104844+0000 2024-02-05T06:22:21.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:22.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:22.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:22:22.623 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:22:22.624 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:22:22.624 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:22.624 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:22.624 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:22:22.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:21:32.104844+0000 '>' 2024-02-05T06:21:32.104844+0000 2024-02-05T06:22:22.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:23.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:23.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:22:23.922 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:22:23.922 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:22:23.922 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:23.923 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:23.923 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:22:24.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:21:32.104844+0000 '>' 2024-02-05T06:21:32.104844+0000 2024-02-05T06:22:24.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:25.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:25.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:22:25.230 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:22:25.230 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:22:25.230 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:25.231 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:25.231 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:22:25.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:22:22.145702+0000 '>' 2024-02-05T06:21:32.104844+0000 2024-02-05T06:22:25.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:22:25.541 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-02-05T06:22:25.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-02-05T06:22:25.542 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-02-05T06:22:25.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-02-05T06:22:25.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-02-05T06:22:25.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-02-05T06:22:25.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-02-05T06:22:25.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-02-05T06:22:25.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-02-05T06:22:25.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:22:25.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:22:25.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:22:25.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:22:25.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:22:25.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:22:25.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:22:25.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:22:25.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:22:25.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:22:25.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:22:26.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:22:26.002 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:22:26.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:22:26.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:22:26.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:22:26.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:22:26.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:22:26.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:22:26.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:22:26.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:22:26.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:22:26.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:22:26.036 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:22:26.036 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:22:26.036 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:22:26.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:22:26.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:22:26.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:22:26.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:22:26.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:22:26.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:22:26.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:22:26.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:22:26.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:22:26.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:22:26.039 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:22:26.039 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:22:26.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:22:26.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:22:26.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:22:26.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:22:26.043 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:22:26.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:22:26.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:22:26.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:22:26.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:22:26.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:22:26.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:22:26.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:22:26.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:22:26.049 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:22:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:22:26.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:22:26.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:22:26.054 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:22:26.054 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:22:26.054 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:22:26.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:22:26.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:22:26.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:22:26.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:22:26.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:22:26.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:22:26.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:22:26.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T06:22:26.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:22:26.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:22:26.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:22:26.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:22:26.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-02-05T06:22:26.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5759: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-02-05T06:22:26.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-02-05T06:22:26.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local objname=POBJ 2024-02-05T06:22:26.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-02-05T06:22:26.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:22:26.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:22:26.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:22:26.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:22:26.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:22:26.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-02-05T06:22:26.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:22:26.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:22:26.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:22:26.516 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:22:26.516 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:22:26.516 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:22:26.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:22:26.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T06:22:26.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:22:26.766 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:22:26.766 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:22:26.766 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:22:26.766 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:22:26.767 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:22:26.767 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:22:26.767 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:22:26.767 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:22:26.768 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:22:26.768 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:22:26.768 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:22:26.768 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:22:26.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:22:26.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T06:22:26.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:22:26.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:22:26.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:22:26.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:22:26.887 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:22:26.887 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:22:26.887 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:22:26.888 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:22:26.888 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:22:26.888 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:22:26.888 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:22:26.888 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:22:26.888 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:22:26.888 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T06:22:26.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5764: TEST_periodic_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-02-05T06:22:26.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:22:26.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:22:26.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:22:26.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:22:26.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:22:26.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:22:28.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:22:28.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:22:28.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:22:28.780 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:22:28.780 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:22:28.780 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:22:28.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:22:28.781 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:22:28.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:22:28.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5765: TEST_periodic_scrub_replicated: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-02-05T06:22:28.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5766: TEST_periodic_scrub_replicated: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-02-05T06:22:28.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5767: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T06:22:28.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:22:28.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:22:28.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:22:28.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:22:28.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:22:28.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:22:28.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:22:28.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:22:28.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:22:28.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:22:28.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:22:28.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:22:28.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:22:28.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:22:28.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:22:28.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:22:28.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:22:28.823 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:22:28.823 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:22:28.823 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:22:28.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:22:28.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:22:28.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:22:28.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:22:28.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:22:28.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:22:28.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:22:28.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:22:28.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:22:28.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:22:28.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:22:28.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:22:28.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-02-05T06:22:28.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:22:28.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:22:28.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=66298e59-28c5-4d5a-84fe-a35ea17fa979 2024-02-05T06:22:28.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 66298e59-28c5-4d5a-84fe-a35ea17fa979' 2024-02-05T06:22:28.826 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 66298e59-28c5-4d5a-84fe-a35ea17fa979 2024-02-05T06:22:28.826 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:22:28.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCkfsBl9540MxAAdB90Gdii64KDn5Y/87klAQ== 2024-02-05T06:22:28.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCkfsBl9540MxAAdB90Gdii64KDn5Y/87klAQ=="}' 2024-02-05T06:22:28.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 66298e59-28c5-4d5a-84fe-a35ea17fa979 -i td/osd-scrub-repair/0/new.json 2024-02-05T06:22:29.137 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:22:29.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:22:29.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQCkfsBl9540MxAAdB90Gdii64KDn5Y/87klAQ== --osd-uuid 66298e59-28c5-4d5a-84fe-a35ea17fa979 2024-02-05T06:22:29.200 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:29.210+0000 7fb5e44b3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:22:29.222 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:29.234+0000 7fb5e44b3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:22:29.223 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:29.234+0000 7fb5e44b3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:22:29.224 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:29.234+0000 7fb5e44b3640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:22:29.224 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:29.234+0000 7fb5e44b3640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:22:29.224 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:29.234+0000 7fb5e44b3640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:22:29.330 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:29.342+0000 7fb5e44b3640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:22:37.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:22:37.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:22:37.682 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:22:37.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:22:37.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:22:38.051 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:22:38.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:22:38.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T06:22:38.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:22:38.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:22:38.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:22:38.086 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:38.098+0000 7f2d161da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:22:38.130 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:38.142+0000 7f2d161da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:22:38.132 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:38.142+0000 7f2d161da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:22:38.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:22:38.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:22:38.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:22:38.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:22:38.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:22:38.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:38.350 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:22:38.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:22:38.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:38.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:38.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:38.713 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:38.722+0000 7f2d161da640 -1 Falling back to public interface 2024-02-05T06:22:39.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:39.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:39.660 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:22:39.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:22:39.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:39.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:39.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:40.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:40.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:40.972 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:22:40.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:22:40.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:40.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:41.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:41.611 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:22:41.622+0000 7f2d161da640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:22:42.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:42.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:42.292 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:22:42.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:22:42.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:42.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:42.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:43.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:43.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:43.740 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:22:43.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:22:43.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:43.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:44.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:45.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:45.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:45.048 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:22:45.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:22:45.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:45.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:45.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:46.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:46.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:46.456 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:22:46.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:22:46.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:46.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:46.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:47.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:47.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:47.924 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:22:47.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:22:47.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:47.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:48.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:49.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:49.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:49.246 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:22:49.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:22:49.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:49.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:49.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:50.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:50.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:50.707 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:22:50.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:22:50.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:50.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:51.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:52.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:52.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:52.019 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:22:52.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:22:52.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:52.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:52.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:53.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:53.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:53.476 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:22:53.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:22:53.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:53.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:53.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:54.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:54.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:54.913 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:22:54.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:22:54.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:54.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:55.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:56.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:56.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:56.221 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:22:56.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:22:56.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:56.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:56.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:57.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:57.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:57.660 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:22:57.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:22:57.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:57.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:57.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:22:58.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:22:58.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:22:58.990 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:22:58.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:22:58.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:22:58.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:22:59.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:00.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:00.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:00.306 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:23:00.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:23:00.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:00.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:23:00.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:01.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:01.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:01.848 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:23:01.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:23:01.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:01.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:23:02.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:03.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:03.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:03.578 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:23:03.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:23:03.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:03.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:23:03.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:04.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:23:04.978 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:23:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:23:05.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:06.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:06.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:06.511 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:23:06.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:23:06.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:06.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:23:07.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:08.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:08.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:08.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T06:23:08.064 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T06:23:08.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:08.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:23:09.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:10.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:10.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:10.055 INFO:tasks.workunit.client.0.smithi148.stdout:22 2024-02-05T06:23:10.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-02-05T06:23:10.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:10.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:23:10.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:11.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:11.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:11.569 INFO:tasks.workunit.client.0.smithi148.stdout:23 2024-02-05T06:23:11.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-02-05T06:23:11.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:11.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:23:12.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:13.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:13.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:13.373 INFO:tasks.workunit.client.0.smithi148.stdout:24 2024-02-05T06:23:13.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-02-05T06:23:13.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:13.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:23:14.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:15.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:15.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:15.019 INFO:tasks.workunit.client.0.smithi148.stdout:25 2024-02-05T06:23:15.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-02-05T06:23:15.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:15.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:23:15.331 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3690937932,v1:127.0.0.1:6803/3690937932] [v2:127.0.0.1:6804/3690937932,v1:127.0.0.1:6805/3690937932] exists,up 66298e59-28c5-4d5a-84fe-a35ea17fa979 2024-02-05T06:23:15.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:23:15.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:23:15.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:23:15.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5768: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T06:23:15.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:23:15.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:23:15.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:23:15.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:23:15.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:23:15.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:23:15.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:23:15.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:23:15.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:23:15.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:23:15.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:23:15.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:23:15.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:23:15.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:23:15.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:23:15.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:23:15.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:23:15.337 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:23:15.337 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:23:15.337 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:23:15.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:23:15.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:23:15.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:23:15.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:23:15.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:23:15.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:23:15.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:23:15.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:23:15.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:23:15.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:23:15.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:23:15.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:23:15.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-02-05T06:23:15.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:23:15.338 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:23:15.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=59fb55eb-50ec-4cdd-b37f-88db006465c5 2024-02-05T06:23:15.339 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 59fb55eb-50ec-4cdd-b37f-88db006465c5 2024-02-05T06:23:15.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 59fb55eb-50ec-4cdd-b37f-88db006465c5' 2024-02-05T06:23:15.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:23:15.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDTfsBlDIYsFhAAoPMBGQfidogKk2SKs5wgdw== 2024-02-05T06:23:15.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDTfsBlDIYsFhAAoPMBGQfidogKk2SKs5wgdw=="}' 2024-02-05T06:23:15.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 59fb55eb-50ec-4cdd-b37f-88db006465c5 -i td/osd-scrub-repair/1/new.json 2024-02-05T06:23:15.940 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:23:15.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T06:23:15.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQDTfsBlDIYsFhAAoPMBGQfidogKk2SKs5wgdw== --osd-uuid 59fb55eb-50ec-4cdd-b37f-88db006465c5 2024-02-05T06:23:16.001 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:23:16.013+0000 7f5e3fbfa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:23:16.075 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:23:16.089+0000 7f5e3fbfa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:23:16.077 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:23:16.089+0000 7f5e3fbfa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:23:16.078 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:23:16.089+0000 7f5e3fbfa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:23:16.078 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:23:16.089+0000 7f5e3fbfa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:23:16.078 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:23:16.089+0000 7f5e3fbfa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:23:16.162 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:23:16.173+0000 7f5e3fbfa640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T06:23:24.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T06:23:24.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:23:24.354 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:23:24.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:23:24.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:23:24.738 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:23:24.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:23:24.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:23:24.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T06:23:24.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:23:24.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:23:24.772 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:23:24.785+0000 7fb18b5d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:23:24.814 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:23:24.825+0000 7fb18b5d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:23:24.816 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:23:24.829+0000 7fb18b5d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:23:25.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:23:25.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:23:25.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:23:25.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:23:25.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:23:25.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:25.033 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:23:25.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:23:25.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:25.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:25.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:25.405 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:23:25.417+0000 7fb18b5d6640 -1 Falling back to public interface 2024-02-05T06:23:26.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:26.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:26.327 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:23:26.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:23:26.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:26.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:26.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:27.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:27.626 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:23:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:23:27.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:27.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:27.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:28.479 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:23:28.489+0000 7fb18b5d6640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:23:28.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:28.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:28.989 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:23:28.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:23:28.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:28.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:29.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:30.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:30.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:30.633 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:23:30.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:23:30.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:30.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:30.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:31.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:31.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:31.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:23:31.952 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:23:31.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:31.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:32.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:33.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:33.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:33.322 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:23:33.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:23:33.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:33.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:33.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:34.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:34.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:34.760 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:23:34.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:23:34.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:34.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:35.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:36.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:36.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:36.074 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:23:36.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:23:36.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:36.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:36.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:37.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:37.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:37.413 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:23:37.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:23:37.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:37.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:37.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:38.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:38.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:38.997 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:23:38.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:23:38.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:38.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:39.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:40.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:40.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:40.366 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:23:40.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:23:40.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:40.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:40.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:41.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:41.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:41.715 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:23:41.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:23:41.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:41.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:42.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:43.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:43.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:43.077 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:23:43.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:23:43.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:43.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:43.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:44.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:44.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:44.401 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:23:44.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:23:44.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:44.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:44.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:45.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:45.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:45.755 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:23:45.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:23:45.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:45.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:46.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:47.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:47.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:47.066 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:23:47.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:23:47.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:47.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:47.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:48.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:48.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:48.442 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:23:48.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:23:48.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:48.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:48.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:49.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:49.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:49.824 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:23:49.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:23:49.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:49.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:51.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:51.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:51.364 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:23:51.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:23:51.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:51.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:51.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:52.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:52.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:52.760 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:23:52.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:23:52.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:52.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:53.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:54.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:54.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:54.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-02-05T06:23:54.315 INFO:tasks.workunit.client.0.smithi148.stdout:21 2024-02-05T06:23:54.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:54.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:55.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:56.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:56.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:56.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-02-05T06:23:56.117 INFO:tasks.workunit.client.0.smithi148.stdout:22 2024-02-05T06:23:56.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:56.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:56.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:57.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:57.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:57.760 INFO:tasks.workunit.client.0.smithi148.stdout:23 2024-02-05T06:23:57.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-02-05T06:23:57.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:57.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:23:58.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:23:59.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:23:59.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:23:59.453 INFO:tasks.workunit.client.0.smithi148.stdout:24 2024-02-05T06:23:59.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-02-05T06:23:59.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:23:59.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:24:00.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:24:01.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:24:01.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:24:01.432 INFO:tasks.workunit.client.0.smithi148.stdout:25 2024-02-05T06:24:01.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-02-05T06:24:01.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:24:01.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:24:01.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:24:02.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:24:02.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:24:02.978 INFO:tasks.workunit.client.0.smithi148.stdout:26 2024-02-05T06:24:02.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-02-05T06:24:02.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:24:02.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:24:03.298 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2826492796,v1:127.0.0.1:6811/2826492796] [v2:127.0.0.1:6812/2826492796,v1:127.0.0.1:6813/2826492796] exists,up 59fb55eb-50ec-4cdd-b37f-88db006465c5 2024-02-05T06:24:03.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:24:03.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:24:03.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:24:03.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5769: TEST_periodic_scrub_replicated: create_rbd_pool 2024-02-05T06:24:03.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T06:24:03.723 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T06:24:03.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T06:24:03.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T06:24:04.433 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T06:24:04.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:24:05.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T06:24:08.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5770: TEST_periodic_scrub_replicated: wait_for_clean 2024-02-05T06:24:08.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:24:08.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:24:08.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:24:08.116 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:24:08.117 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:24:08.117 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:24:08.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:24:08.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:24:08.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:24:08.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:24:08.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:24:08.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:24:08.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:24:08.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:24:08.256 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:24:08.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:24:08.585 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:24:08.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:24:08.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:24:08.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:24:08.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-02-05T06:24:08.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-02-05T06:24:08.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-02-05T06:24:08.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:24:08.732 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:24:08.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-02-05T06:24:08.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-02-05T06:24:08.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672969' 2024-02-05T06:24:08.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:24:08.875 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-02-05T06:24:08.876 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:24:08.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:24:08.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-02-05T06:24:08.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:24:08.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-02-05T06:24:08.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-02-05T06:24:08.879 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836497 2024-02-05T06:24:08.879 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:24:09.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-02-05T06:24:09.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:24:10.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:24:10.193 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:24:10.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-02-05T06:24:10.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:24:10.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-02-05T06:24:10.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:24:10.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:24:10.499 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-02-05T06:24:10.499 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:24:10.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-02-05T06:24:10.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-02-05T06:24:10.501 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672969 2024-02-05T06:24:10.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:24:10.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672969 2024-02-05T06:24:10.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:24:10.793 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:24:10.793 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:24:11.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T06:24:11.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:24:11.167 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:24:11.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:24:11.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:24:11.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:24:11.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:24:11.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:24:11.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:24:11.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:24:11.476 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:24:11.476 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:24:11.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T06:24:11.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:24:11.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:24:11.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5772: TEST_periodic_scrub_replicated: create_pool psr_pool 1 1 2024-02-05T06:24:11.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 1 1 2024-02-05T06:24:12.256 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'psr_pool' created 2024-02-05T06:24:12.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:24:13.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5773: TEST_periodic_scrub_replicated: wait_for_clean 2024-02-05T06:24:13.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:24:13.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:24:13.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:24:13.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:24:13.278 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:24:13.278 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:24:13.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:24:13.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:24:13.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:24:13.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:24:13.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:24:13.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:24:13.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:24:13.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:24:13.417 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:24:13.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:24:13.846 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:24:13.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:24:13.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:24:13.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:24:14.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-02-05T06:24:14.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-02-05T06:24:14.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-02-05T06:24:14.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:24:14.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:24:14.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-02-05T06:24:14.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-02-05T06:24:14.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672971' 2024-02-05T06:24:14.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:24:14.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-02-05T06:24:14.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:24:14.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:24:14.249 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-02-05T06:24:14.249 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:24:14.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-02-05T06:24:14.251 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836499 2024-02-05T06:24:14.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-02-05T06:24:14.251 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:24:14.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-02-05T06:24:14.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:24:15.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:24:15.647 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:24:15.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836499 2024-02-05T06:24:15.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:24:15.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-02-05T06:24:15.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:24:15.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:24:15.979 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-02-05T06:24:15.979 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:24:15.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-02-05T06:24:15.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-02-05T06:24:15.981 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672971 2024-02-05T06:24:15.981 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:24:16.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-02-05T06:24:16.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:24:16.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:24:16.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:24:16.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T06:24:16.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:24:16.687 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:24:16.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:24:16.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:24:16.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:24:16.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:24:16.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:24:17.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T06:24:17.020 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:24:17.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:24:17.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:24:17.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T06:24:17.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:24:17.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:24:17.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5775: TEST_periodic_scrub_replicated: local osd=0 2024-02-05T06:24:17.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5776: TEST_periodic_scrub_replicated: add_something td/osd-scrub-repair psr_pool POBJ scrub 2024-02-05T06:24:17.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:24:17.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-02-05T06:24:17.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-02-05T06:24:17.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=scrub 2024-02-05T06:24:17.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' scrub = noscrub ']' 2024-02-05T06:24:17.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:81: add_something: ceph osd unset noscrub 2024-02-05T06:24:17.820 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is unset 2024-02-05T06:24:17.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: ceph osd unset nodeep-scrub 2024-02-05T06:24:18.355 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is unset 2024-02-05T06:24:18.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:24:18.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:24:18.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-02-05T06:24:18.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: get_primary psr_pool POBJ 2024-02-05T06:24:18.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-02-05T06:24:18.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-02-05T06:24:18.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:24:18.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-02-05T06:24:19.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: local primary=0 2024-02-05T06:24:19.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: get_pg psr_pool POBJ 2024-02-05T06:24:19.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-02-05T06:24:19.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-02-05T06:24:19.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:24:19.012 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-02-05T06:24:19.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: local pg=2.0 2024-02-05T06:24:19.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5781: TEST_periodic_scrub_replicated: local payload=UVWXYZ 2024-02-05T06:24:19.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5782: TEST_periodic_scrub_replicated: echo UVWXYZ 2024-02-05T06:24:19.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5784: TEST_periodic_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:24:19.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:24:19.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:24:19.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:24:19.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:24:19.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:24:19.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:24:19.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:24:19.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:24:19.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:24:19.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:24:19.331 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:24:19.332 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:24:19.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:24:19.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:24:19.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:24:20.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:24:20.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:24:20.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:24:20.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:24:20.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:24:20.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:24:20.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:24:20.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T06:24:22.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T06:24:22.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:24:22.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:24:22.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:24:22.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:24:22.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:24:22.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:24:22.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:24:22.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:24:22.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:24:22.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:24:22.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:24:22.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:24:22.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:24:22.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:24:22.998 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:24:22.998 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:24:22.998 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:24:22.999 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:24:23.000 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:24:23.000 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:24:23.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:24:23.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:24:23.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:24:23.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:24:23.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:24:23.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:24:23.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:24:23.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:24:23.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:24:23.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:24:23.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-02-05T06:24:23.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:24:23.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:24:23.001 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:24:23.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T06:24:23.002 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:24:23.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:24:23.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:24:23.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:24:23.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:24:23.037 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:24:23.048+0000 7fd196ef7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:23.040 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:24:23.048+0000 7fd196ef7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:23.042 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:24:23.052+0000 7fd196ef7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:23.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:24:23.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:24:23.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:24:23.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:24:23.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:24:23.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:24:23.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:24:23.310 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:24:23.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:24:23.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:24:23.584 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:24:23.592+0000 7fd196ef7640 -1 Falling back to public interface 2024-02-05T06:24:23.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:24:24.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:24:24.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:24:24.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:24:24.614 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:24:24.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:24:24.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:24:24.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:24:25.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:24:25.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:24:25.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:24:25.919 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:24:25.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:24:25.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:24:26.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:24:26.661 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:24:26.667+0000 7fd196ef7640 -1 osd.0 24 log_to_monitors true 2024-02-05T06:24:27.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:24:27.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:24:27.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:24:27.241 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:24:27.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:24:27.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:24:27.786 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 28 up_thru 19 down_at 25 last_clean_interval [5,24) [v2:127.0.0.1:6802/2112936699,v1:127.0.0.1:6803/2112936699] [v2:127.0.0.1:6804/2112936699,v1:127.0.0.1:6805/2112936699] exists,up 66298e59-28c5-4d5a-84fe-a35ea17fa979 2024-02-05T06:24:27.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:24:27.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:24:27.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:24:27.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:24:27.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:24:27.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:24:27.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:24:27.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:24:27.788 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:24:27.788 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:24:27.789 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:24:27.789 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:24:27.789 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:24:27.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:24:27.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:24:27.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:24:27.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:24:27.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:24:27.908 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:24:28.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:24:28.503 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:24:28.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:24:28.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:24:28.504 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:24:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084290 2024-02-05T06:24:28.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084290 2024-02-05T06:24:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084290' 2024-02-05T06:24:28.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:24:28.653 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:24:28.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-02-05T06:24:28.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-02-05T06:24:28.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084290 1-42949672975' 2024-02-05T06:24:28.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:24:28.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084290 2024-02-05T06:24:28.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:24:28.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:24:28.794 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084290 2024-02-05T06:24:28.794 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:24:28.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084290 2024-02-05T06:24:28.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084290' 2024-02-05T06:24:28.795 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 120259084290 2024-02-05T06:24:28.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:24:29.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 120259084290 2024-02-05T06:24:29.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:24:30.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:24:30.107 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:24:30.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084290 -lt 120259084290 2024-02-05T06:24:30.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:24:30.431 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-02-05T06:24:30.431 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:24:30.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:24:30.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-02-05T06:24:30.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:24:30.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-02-05T06:24:30.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-02-05T06:24:30.435 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672975 2024-02-05T06:24:30.436 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:24:30.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-02-05T06:24:30.786 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:24:30.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:24:30.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:24:31.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T06:24:31.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:24:31.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:24:31.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:24:31.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:24:31.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:24:31.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:24:31.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:24:31.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T06:24:31.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:24:31.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:24:31.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:24:31.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T06:24:31.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:24:31.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:24:31.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5787: TEST_periodic_scrub_replicated: set -o pipefail 2024-02-05T06:24:31.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-02-05T06:24:31.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: jq . 2024-02-05T06:24:31.888 INFO:tasks.workunit.client.0.smithi148.stderr:No scrub information available for pg 2.0 2024-02-05T06:24:31.888 INFO:tasks.workunit.client.0.smithi148.stderr:error 2: (2) No such file or directory 2024-02-05T06:24:31.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5789: TEST_periodic_scrub_replicated: set +o pipefail 2024-02-05T06:24:31.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5791: TEST_periodic_scrub_replicated: pg_deep_scrub 2.0 2024-02-05T06:24:31.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=2.0 2024-02-05T06:24:31.893 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-02-05T06:24:31.893 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:24:31.893 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:24:31.893 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:31.893 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-02-05T06:24:32.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-02-05T06:24:12.188857+0000 2024-02-05T06:24:32.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 2.0 2024-02-05T06:24:32.483 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0 on osd.0 to deep-scrub 2024-02-05T06:24:32.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 2.0 2024-02-05T06:24:12.188857+0000 last_deep_scrub_stamp 2024-02-05T06:24:32.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T06:24:32.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:24:12.188857+0000 2024-02-05T06:24:32.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-02-05T06:24:32.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:24:32.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:24:32.501 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-02-05T06:24:32.501 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:24:32.501 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:24:32.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:32.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-02-05T06:24:32.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:24:12.188857+0000 '>' 2024-02-05T06:24:12.188857+0000 2024-02-05T06:24:32.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:33.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:33.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:24:33.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-02-05T06:24:33.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:24:33.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:24:33.805 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:33.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-02-05T06:24:34.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:24:12.188857+0000 '>' 2024-02-05T06:24:12.188857+0000 2024-02-05T06:24:34.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:35.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:35.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:24:35.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-02-05T06:24:35.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:24:35.111 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:24:35.111 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:35.111 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-02-05T06:24:35.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:24:12.188857+0000 '>' 2024-02-05T06:24:12.188857+0000 2024-02-05T06:24:35.409 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:36.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:36.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:24:36.411 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-02-05T06:24:36.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:24:36.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:24:36.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:36.413 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-02-05T06:24:36.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:24:12.188857+0000 '>' 2024-02-05T06:24:12.188857+0000 2024-02-05T06:24:36.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:37.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:37.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:24:37.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-02-05T06:24:37.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:24:37.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T06:24:37.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:37.735 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-02-05T06:24:38.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:24:32.753171+0000 '>' 2024-02-05T06:24:12.188857+0000 2024-02-05T06:24:38.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:24:38.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-02-05T06:24:38.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: jq . 2024-02-05T06:24:38.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: grep -q POBJ 2024-02-05T06:24:38.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5796: TEST_periodic_scrub_replicated: flush_pg_stats 2024-02-05T06:24:38.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:24:38.072 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:24:38.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:24:38.362 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:24:38.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:24:38.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:24:38.362 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:24:38.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084293 2024-02-05T06:24:38.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084293 2024-02-05T06:24:38.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084293' 2024-02-05T06:24:38.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:24:38.508 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:24:38.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-02-05T06:24:38.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-02-05T06:24:38.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084293 1-42949672978' 2024-02-05T06:24:38.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:24:38.642 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084293 2024-02-05T06:24:38.642 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:24:38.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:24:38.644 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084293 2024-02-05T06:24:38.644 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:24:38.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084293 2024-02-05T06:24:38.645 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 120259084293 2024-02-05T06:24:38.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084293' 2024-02-05T06:24:38.646 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:24:38.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084292 -lt 120259084293 2024-02-05T06:24:38.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:24:39.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:24:39.958 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:24:40.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084293 -lt 120259084293 2024-02-05T06:24:40.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:24:40.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-02-05T06:24:40.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:24:40.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:24:40.270 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-02-05T06:24:40.270 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:24:40.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-02-05T06:24:40.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-02-05T06:24:40.272 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672978 2024-02-05T06:24:40.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:24:40.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-02-05T06:24:40.633 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: get_last_scrub_stamp 2.0 2024-02-05T06:24:40.633 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:24:40.633 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:40.633 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:40.634 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:24:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: local last_scrub=2024-02-05T06:24:32.753171+0000 2024-02-05T06:24:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-02-05T06:24:41.127 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:24:41.127 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:24:41.127 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:24:41.127 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-29T06:23:01.138963+0000" 2024-02-05T06:24:41.127 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:24:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5801: TEST_periodic_scrub_replicated: wait_for_scrub 2.0 2024-02-05T06:24:32.753171+0000 2024-02-05T06:24:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T06:24:41.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:24:32.753171+0000 2024-02-05T06:24:41.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:24:41.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:24:41.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:24:41.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:24:41.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:24:41.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:41.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:24:41.143 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:41.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:24:32.753171+0000 '>' 2024-02-05T06:24:32.753171+0000 2024-02-05T06:24:41.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:42.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:42.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:24:42.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:24:42.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:24:42.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:42.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:24:42.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:42.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:24:32.753171+0000 '>' 2024-02-05T06:24:32.753171+0000 2024-02-05T06:24:42.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:43.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:43.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:24:43.774 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:24:43.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:24:43.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:43.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:24:43.775 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:44.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:24:41.768298+0000 '>' 2024-02-05T06:24:32.753171+0000 2024-02-05T06:24:44.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:24:44.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5804: TEST_periodic_scrub_replicated: grep -q 'Deep scrub errors, upgrading scrub to deep-scrub' td/osd-scrub-repair/osd.0.log 2024-02-05T06:24:44.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-02-05T06:24:44.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: jq . 2024-02-05T06:24:44.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: grep -q POBJ 2024-02-05T06:24:44.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5810: TEST_periodic_scrub_replicated: ceph osd set nodeep-scrub 2024-02-05T06:24:44.578 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:24:44.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5812: TEST_periodic_scrub_replicated: ceph tell osd.0 get_latest_osdmap 2024-02-05T06:24:44.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5813: TEST_periodic_scrub_replicated: flush_pg_stats 2024-02-05T06:24:44.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:24:44.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:24:45.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:24:45.050 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:24:45.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:24:45.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:24:45.050 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:24:45.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084295 2024-02-05T06:24:45.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084295 2024-02-05T06:24:45.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084295' 2024-02-05T06:24:45.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:24:45.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:24:45.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-02-05T06:24:45.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-02-05T06:24:45.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084295 1-42949672980' 2024-02-05T06:24:45.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:24:45.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084295 2024-02-05T06:24:45.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:24:45.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:24:45.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084295 2024-02-05T06:24:45.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:24:45.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084295 2024-02-05T06:24:45.332 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 120259084295 2024-02-05T06:24:45.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084295' 2024-02-05T06:24:45.332 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:24:45.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084295 -lt 120259084295 2024-02-05T06:24:45.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:24:45.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-02-05T06:24:45.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:24:45.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:24:45.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-02-05T06:24:45.623 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:24:45.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-02-05T06:24:45.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-02-05T06:24:45.624 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672980 2024-02-05T06:24:45.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:24:45.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672980 2024-02-05T06:24:45.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:24:46.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:24:46.950 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:24:47.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672980 2024-02-05T06:24:47.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:24:48.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:24:48.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:24:48.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-02-05T06:24:48.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5814: TEST_periodic_scrub_replicated: sleep 5 2024-02-05T06:24:53.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5817: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-02-05T06:24:53.721 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:24:53.721 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:24:53.721 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:24:53.722 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-29T06:23:13.733243+0000" 2024-02-05T06:24:53.722 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:24:53.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5820: TEST_periodic_scrub_replicated: local found=false 2024-02-05T06:24:53.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: seq 14 -1 0 2024-02-05T06:24:53.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: for i in $(seq 14 -1 0) 2024-02-05T06:24:53.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5823: TEST_periodic_scrub_replicated: sleep 1 2024-02-05T06:24:54.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: grep -q 'Regular scrub skipped due to deep-scrub errors and nodeep-scrub set' td/osd-scrub-repair/osd.0.log 2024-02-05T06:24:54.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: found=true 2024-02-05T06:24:54.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: break 2024-02-05T06:24:54.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5827: TEST_periodic_scrub_replicated: test true = true 2024-02-05T06:24:54.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-02-05T06:24:54.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: jq . 2024-02-05T06:24:54.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: grep -q POBJ 2024-02-05T06:24:54.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5832: TEST_periodic_scrub_replicated: flush_pg_stats 2024-02-05T06:24:54.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:24:54.786 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:24:55.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:24:55.073 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:24:55.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:24:55.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:24:55.073 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:24:55.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084298 2024-02-05T06:24:55.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084298 2024-02-05T06:24:55.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084298' 2024-02-05T06:24:55.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:24:55.268 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:24:55.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-02-05T06:24:55.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-02-05T06:24:55.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084298 1-42949672983' 2024-02-05T06:24:55.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:24:55.429 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084298 2024-02-05T06:24:55.429 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:24:55.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:24:55.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084298 2024-02-05T06:24:55.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:24:55.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084298 2024-02-05T06:24:55.433 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 120259084298 2024-02-05T06:24:55.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084298' 2024-02-05T06:24:55.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:24:55.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084298 -lt 120259084298 2024-02-05T06:24:55.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:24:55.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-02-05T06:24:55.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:24:55.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:24:55.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-02-05T06:24:55.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:24:55.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-02-05T06:24:55.735 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672983 2024-02-05T06:24:55.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-02-05T06:24:55.735 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:24:56.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672983 2024-02-05T06:24:56.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:24:57.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:24:57.035 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:24:57.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672983 2024-02-05T06:24:57.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:24:58.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:24:58.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:24:58.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-02-05T06:24:58.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5834: TEST_periodic_scrub_replicated: pg_schedule_scrub 2.0 2024-02-05T06:24:58.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: pg_schedule_scrub: local pgid=2.0 2024-02-05T06:24:58.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: get_last_scrub_stamp 2.0 2024-02-05T06:24:58.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:24:58.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:58.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:58.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:24:59.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: local last_scrub=2024-01-29T06:23:13.733243+0000 2024-02-05T06:24:59.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: pg_schedule_scrub: ceph pg scrub 2.0 2024-02-05T06:24:59.305 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 2.0 on osd.0 to scrub 2024-02-05T06:24:59.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: pg_schedule_scrub: wait_for_scrub 2.0 2024-01-29T06:23:13.733243+0000 2024-02-05T06:24:59.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T06:24:59.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-01-29T06:23:13.733243+0000 2024-02-05T06:24:59.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:24:59.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:24:59.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:24:59.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:24:59.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:24:59.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:59.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:59.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:24:59.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-29T06:23:13.733243+0000 '>' 2024-01-29T06:23:13.733243+0000 2024-02-05T06:24:59.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:25:00.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:25:00.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:25:00.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:25:00.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:25:00.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:25:00.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:25:00.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:25:00.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-29T06:23:13.733243+0000 '>' 2024-01-29T06:23:13.733243+0000 2024-02-05T06:25:00.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:25:01.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:25:01.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:25:01.942 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:25:01.943 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:25:01.943 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:25:01.943 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:25:01.943 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:25:02.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-29T06:23:13.733243+0000 '>' 2024-01-29T06:23:13.733243+0000 2024-02-05T06:25:02.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:25:03.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:25:03.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:25:03.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T06:25:03.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T06:25:03.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:25:03.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:25:03.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T06:25:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:24:59.688265+0000 '>' 2024-01-29T06:23:13.733243+0000 2024-02-05T06:25:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:25:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5835: TEST_periodic_scrub_replicated: grep -q 'Regular scrub request, deep-scrub details will be lost' td/osd-scrub-repair/osd.0.log 2024-02-05T06:25:03.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-02-05T06:25:03.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: jq . 2024-02-05T06:25:03.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: grep -qv POBJ 2024-02-05T06:25:03.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:25:03.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:25:03.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:25:03.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:25:03.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:25:03.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:25:03.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:25:03.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:25:03.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:25:03.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:25:03.949 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:25:03.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:25:03.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:25:03.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:25:03.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:25:03.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:25:03.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:25:03.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:25:03.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:25:03.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:25:03.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:25:03.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:25:03.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:25:03.979 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:25:03.980 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:03.980 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:25:03.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:25:03.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:25:03.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:25:03.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:25:03.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:25:03.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:25:03.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:25:03.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:25:03.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:25:03.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:25:03.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:25:03.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:25:03.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:25:03.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:25:03.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:25:03.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:25:03.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:25:03.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:25:03.988 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:25:03.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:25:03.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:25:03.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:25:03.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:25:03.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:25:03.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:25:03.992 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:25:03.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:25:03.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:25:03.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:25:03.997 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:25:03.997 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:03.997 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:25:03.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:25:03.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:25:03.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:25:03.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:25:04.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:25:04.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:04.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:25:04.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T06:25:04.002 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:25:04.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:25:04.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:25:04.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:25:04.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-02-05T06:25:04.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-02-05T06:25:04.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-02-05T06:25:04.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-02-05T06:25:04.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-02-05T06:25:04.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-02-05T06:25:04.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-02-05T06:25:04.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:25:04.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:25:04.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:25:04.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:25:04.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:25:04.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T06:25:04.409 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:25:04.409 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:25:04.409 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:25:04.410 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:25:04.410 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:04.410 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:25:04.410 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:25:04.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T06:25:04.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:25:04.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:25:04.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:25:04.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:25:04.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:25:04.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:25:04.620 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:25:04.620 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:25:04.621 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:25:04.621 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:25:04.621 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:04.621 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:25:04.622 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:25:04.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:25:04.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T06:25:04.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:25:04.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:25:04.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:25:04.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:25:04.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:25:04.741 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:25:04.741 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:25:04.741 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:25:04.741 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:25:04.742 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:04.742 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:25:04.742 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:25:04.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:25:04.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T06:25:04.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-02-05T06:25:04.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:25:04.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:25:04.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:25:04.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:25:04.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:25:04.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:25:05.728 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:25:05.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:25:05.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:25:05.729 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:25:05.729 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:05.729 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:25:05.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:25:05.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:25:05.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:25:05.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T06:25:05.766 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-02-05T06:25:05.767 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-02-05T06:25:05.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:25:05.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:25:05.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:25:05.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:25:05.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:25:05.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:25:05.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:25:05.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:25:05.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:25:05.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:25:05.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:25:05.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:25:05.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:25:05.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:25:05.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:25:05.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:25:05.772 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:25:05.772 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:25:05.772 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:25:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:25:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:25:05.772 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:25:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:25:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:25:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:25:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:25:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:25:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:25:05.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:25:05.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:25:05.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:25:05.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:25:05.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:25:05.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T06:25:05.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:25:05.773 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:25:05.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0e051bd0-257b-438f-b9ca-f0ebe6859f71 2024-02-05T06:25:05.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0e051bd0-257b-438f-b9ca-f0ebe6859f71' 2024-02-05T06:25:05.774 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 0e051bd0-257b-438f-b9ca-f0ebe6859f71 2024-02-05T06:25:05.774 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:25:05.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBf8Blj+i/LxAAOWk4y/xH72dzhtZTeQiILA== 2024-02-05T06:25:05.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBf8Blj+i/LxAAOWk4y/xH72dzhtZTeQiILA=="}' 2024-02-05T06:25:05.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0e051bd0-257b-438f-b9ca-f0ebe6859f71 -i td/osd-scrub-repair/0/new.json 2024-02-05T06:25:06.065 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:25:06.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:25:06.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBBf8Blj+i/LxAAOWk4y/xH72dzhtZTeQiILA== --osd-uuid 0e051bd0-257b-438f-b9ca-f0ebe6859f71 2024-02-05T06:25:06.122 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:06.127+0000 7f4b7628b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:06.158 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:06.163+0000 7f4b7628b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:06.160 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:06.167+0000 7f4b7628b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:06.161 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:06.167+0000 7f4b7628b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:25:06.161 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:06.167+0000 7f4b7628b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:25:06.161 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:06.167+0000 7f4b7628b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:25:06.258 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:06.263+0000 7f4b7628b640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:25:12.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:25:12.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:25:12.331 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:25:12.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:25:12.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:25:12.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:25:12.723 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:25:12.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:25:12.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:25:12.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:25:12.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:25:12.757 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:12.762+0000 7f548eb3c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:12.791 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:12.798+0000 7f548eb3c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:12.793 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:12.798+0000 7f548eb3c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:13.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:25:13.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:25:13.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:25:13.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:25:13.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:25:13.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:13.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:25:13.046 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:25:13.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:13.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:13.362 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:13.366+0000 7f548eb3c640 -1 Falling back to public interface 2024-02-05T06:25:13.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:14.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:14.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:14.386 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:25:14.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:25:14.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:14.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:14.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:15.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:15.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:15.711 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:25:15.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:25:15.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:15.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:15.883 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:15.890+0000 7f548eb3c640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:25:16.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:17.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:17.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:17.077 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:25:17.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:25:17.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:17.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:17.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:18.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:18.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:18.431 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:25:18.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:25:18.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:18.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:18.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:19.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:19.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:19.856 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:25:19.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:25:19.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:19.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:20.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:21.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:21.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:21.166 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:25:21.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:25:21.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:21.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:21.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:22.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:22.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:22.487 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:25:22.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:25:22.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:22.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:22.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:23.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:23.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:23.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:25:23.871 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:25:23.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:23.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:24.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:25.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:25.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:25.185 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:25:25.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:25:25.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:25.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:25.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:26.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:26.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:26.492 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:25:26.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:25:26.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:26.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:26.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:27.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:27.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:27.843 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:25:27.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:25:27.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:27.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:28.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:29.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:29.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:29.164 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:25:29.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:25:29.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:29.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:29.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:30.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:30.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:30.645 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:25:30.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:25:30.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:30.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:31.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:32.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:32.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:32.164 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:25:32.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:25:32.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:32.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:33.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:34.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:34.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:34.018 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:25:34.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:25:34.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:34.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:34.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:35.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:35.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:35.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:25:35.864 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:25:35.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:35.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:36.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:37.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:37.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:37.511 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:25:37.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:25:37.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:37.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:38.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:39.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:39.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:39.339 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:25:39.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:25:39.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:39.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:39.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:40.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:40.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:40.688 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:25:40.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:25:40.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:40.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:40.987 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4206742983,v1:127.0.0.1:6803/4206742983] [v2:127.0.0.1:6804/4206742983,v1:127.0.0.1:6805/4206742983] exists,up 0e051bd0-257b-438f-b9ca-f0ebe6859f71 2024-02-05T06:25:40.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:25:40.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:25:40.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:25:40.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:25:40.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:25:40.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:25:40.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:25:40.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:25:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:25:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:25:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:25:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:25:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:25:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:25:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:25:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:25:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:25:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:25:40.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:25:40.992 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:25:40.992 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:25:40.993 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:25:40.993 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:25:40.993 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:40.993 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:25:40.993 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:25:40.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:25:40.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:25:40.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:25:40.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:25:40.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:25:40.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:25:40.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:25:40.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:25:40.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:25:40.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:25:40.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:25:40.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T06:25:40.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:25:40.994 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:25:40.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=15d0d83a-e44a-49a4-9cfe-b1413755c6e6 2024-02-05T06:25:40.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 15d0d83a-e44a-49a4-9cfe-b1413755c6e6' 2024-02-05T06:25:40.995 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 15d0d83a-e44a-49a4-9cfe-b1413755c6e6 2024-02-05T06:25:40.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:25:41.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlf8BlVqg+ARAAiIx9h8rglzKgnkKv/05ngw== 2024-02-05T06:25:41.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlf8BlVqg+ARAAiIx9h8rglzKgnkKv/05ngw=="}' 2024-02-05T06:25:41.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 15d0d83a-e44a-49a4-9cfe-b1413755c6e6 -i td/osd-scrub-repair/1/new.json 2024-02-05T06:25:41.395 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:25:41.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T06:25:41.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBlf8BlVqg+ARAAiIx9h8rglzKgnkKv/05ngw== --osd-uuid 15d0d83a-e44a-49a4-9cfe-b1413755c6e6 2024-02-05T06:25:41.445 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:41.454+0000 7f0a47adf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:41.480 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:41.490+0000 7f0a47adf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:41.482 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:41.490+0000 7f0a47adf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:41.482 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:41.490+0000 7f0a47adf640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:25:41.482 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:41.490+0000 7f0a47adf640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:25:41.482 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:41.490+0000 7f0a47adf640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:25:41.556 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:41.566+0000 7f0a47adf640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T06:25:47.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T06:25:47.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:25:47.999 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:25:47.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:25:47.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:25:48.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:25:48.377 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:25:48.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:25:48.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:25:48.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:25:48.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:25:48.410 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:48.418+0000 7fd9902d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:48.411 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:48.418+0000 7fd9902d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:48.413 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:48.422+0000 7fd9902d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:48.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:25:48.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:25:48.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:25:48.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:25:48.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:25:48.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:48.720 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:25:48.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:25:48.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:48.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:25:48.946 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:48.954+0000 7fd9902d3640 -1 Falling back to public interface 2024-02-05T06:25:49.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:50.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:50.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:50.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:25:50.168 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:25:50.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:50.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:25:50.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:51.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:51.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:51.484 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:25:51.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:25:51.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:51.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:25:51.578 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:25:51.585+0000 7fd9902d3640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:25:51.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:52.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:52.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:52.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:25:52.789 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:25:52.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:52.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:25:53.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:54.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:54.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:54.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:25:54.133 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:25:54.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:54.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:25:54.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:55.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:55.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:55.687 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:25:55.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:25:55.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:55.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:25:56.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:57.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:57.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:57.010 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:25:57.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:25:57.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:57.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:25:57.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:58.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:58.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:58.450 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:25:58.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:25:58.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:58.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:25:58.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:59.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:59.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:25:59.765 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:25:59.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:25:59.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:59.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:00.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:01.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:01.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:26:01.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:26:01.083 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:26:01.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:01.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:01.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:02.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:02.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:26:02.435 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:26:02.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:26:02.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:02.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:02.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:03.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:03.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:26:03.747 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:26:03.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:26:03.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:03.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:04.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:05.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:05.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:26:05.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:26:05.053 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:26:05.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:05.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:05.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:06.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:06.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:26:06.571 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:26:06.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:26:06.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:06.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:07.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:08.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:08.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:26:08.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:26:08.349 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:26:08.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:08.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:09.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:10.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:10.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:26:10.407 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:26:10.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:26:10.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:10.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:10.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:11.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:11.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:26:11.832 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:26:11.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:26:11.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:11.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:12.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:13.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:13.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:26:13.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:26:13.292 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:26:13.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:13.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:14.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:15.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:15.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:26:15.126 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:26:15.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:26:15.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:15.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:15.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:16.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:16.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:26:16.448 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:26:16.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:26:16.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:16.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:16.762 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3772779258,v1:127.0.0.1:6811/3772779258] [v2:127.0.0.1:6812/3772779258,v1:127.0.0.1:6813/3772779258] exists,up 15d0d83a-e44a-49a4-9cfe-b1413755c6e6 2024-02-05T06:26:16.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:26:16.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:26:16.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:26:16.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-02-05T06:26:16.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-02-05T06:26:17.201 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'testpool' created 2024-02-05T06:26:17.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:26:18.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-02-05T06:26:18.630 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 1 size to 2 2024-02-05T06:26:18.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-02-05T06:26:18.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:26:18.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:26:18.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:26:18.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:26:18.653 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:26:18.653 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:26:18.653 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:26:18.653 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:26:18.653 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:26:18.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:26:18.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:26:18.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:26:18.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:26:18.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:26:18.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:26:19.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:26:19.135 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:26:19.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:26:19.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:26:19.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:26:19.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-02-05T06:26:19.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-02-05T06:26:19.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-02-05T06:26:19.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:26:19.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:26:19.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-02-05T06:26:19.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-02-05T06:26:19.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672966' 2024-02-05T06:26:19.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:26:19.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-02-05T06:26:19.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:26:19.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:26:19.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-02-05T06:26:19.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:26:19.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-02-05T06:26:19.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-02-05T06:26:19.476 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836493 2024-02-05T06:26:19.477 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:26:19.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-02-05T06:26:19.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:26:20.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:26:20.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:26:21.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-02-05T06:26:21.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:26:21.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-02-05T06:26:21.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:26:21.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:26:21.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-02-05T06:26:21.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:26:21.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-02-05T06:26:21.140 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672966 2024-02-05T06:26:21.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-02-05T06:26:21.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:26:21.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672966 2024-02-05T06:26:21.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:26:21.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:26:21.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:26:21.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:26:21.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:26:21.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:26:21.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:26:21.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:26:21.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:26:21.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:26:21.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:26:22.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:26:22.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:26:22.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:26:22.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:26:22.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:26:22.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:26:22.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:26:22.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-02-05T06:26:22.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-02-05T06:26:22.548 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-02-05T06:26:22.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:22.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:22.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:22.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:22.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:22.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:22.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:22.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:23.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:23.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:23.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:23.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:23.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:23.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:23.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:23.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:23.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:23.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:23.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:23.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:24.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:24.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:24.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:24.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:24.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:24.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:24.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:24.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:24.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:24.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:24.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:24.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:25.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:25.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:25.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:25.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:25.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:25.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:25.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:25.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:25.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:25.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:25.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:25.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:25.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:25.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:26.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:26.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:26.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:26.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:26.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:26.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:26.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:26.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:26.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:26.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:26.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:26.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:26.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T06:26:26.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:26:27.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-02-05T06:26:27.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T06:26:27.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-02-05T06:26:27.102 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-02-05T06:26:27.102 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T06:26:27.102 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:26:27.102 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-02-05T06:26:27.103 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:26:27.421 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T06:26:27.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-02-05T06:26:27.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T06:26:27.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-02-05T06:26:27.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-02-05T06:26:27.756 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-02-05T06:26:27.756 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-02-05T06:26:27.756 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:26:27.757 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-02-05T06:26:28.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-02-05T06:26:28.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-02-05T06:26:28.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T06:26:28.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:26:28.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-02-05T06:26:28.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:26:28.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-02-05T06:26:28.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:26:28.393 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:26:28.393 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:26:28.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:26:28.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:26:28.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:26:28.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:26:28.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:26:28.708 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:26:28.708 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:26:28.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:26:28.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:26:28.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:26:30.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:26:30.022 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-02-05T06:26:30.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:30.025 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-02-05T06:26:30.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T06:26:30.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-02-05T06:26:30.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:30.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:30.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:26:30.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:30.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:26:30.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-02-05T06:26:31.368 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:ff7b1f36:::obj1:head# 2024-02-05T06:26:31.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:31.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-02-05T06:26:31.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T06:26:31.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-02-05T06:26:31.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:31.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:31.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:26:31.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:31.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:26:31.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-02-05T06:26:33.178 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:104778fc:::obj2:head# 2024-02-05T06:26:33.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:33.712 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-02-05T06:26:33.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T06:26:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-02-05T06:26:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:26:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:26:33.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-02-05T06:26:34.667 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:8dd16f86:::obj3:head# 2024-02-05T06:26:35.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:35.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-02-05T06:26:35.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T06:26:35.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-02-05T06:26:35.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:35.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:35.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:26:35.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:35.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:26:35.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-02-05T06:26:36.220 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:0ee9ae15:::obj4:head# 2024-02-05T06:26:36.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:36.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-02-05T06:26:36.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T06:26:36.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-02-05T06:26:36.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:36.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:36.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:26:36.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:36.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:26:36.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-02-05T06:26:37.489 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:c52c9666:::obj5:head# 2024-02-05T06:26:38.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:38.032 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-02-05T06:26:38.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T06:26:38.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-02-05T06:26:38.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:38.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:38.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:26:38.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:38.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:26:38.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-02-05T06:26:38.733 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:233a42c1:::obj6:head# 2024-02-05T06:26:39.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:39.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-02-05T06:26:39.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T06:26:39.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-02-05T06:26:39.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:39.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:39.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:26:39.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:39.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:26:39.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-02-05T06:26:39.968 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:9a09113a:::obj7:head# 2024-02-05T06:26:40.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:40.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-02-05T06:26:40.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T06:26:40.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-02-05T06:26:40.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:40.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:40.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:26:40.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:40.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:26:40.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-02-05T06:26:41.388 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:a557efb1:::obj8:head# 2024-02-05T06:26:41.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:41.932 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-02-05T06:26:41.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T06:26:41.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-02-05T06:26:41.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:41.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:41.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:26:41.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:41.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:26:41.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-02-05T06:26:42.689 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:fe351e27:::obj9:head# 2024-02-05T06:26:43.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:43.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-02-05T06:26:43.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T06:26:43.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-02-05T06:26:43.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:43.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:43.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:26:43.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:43.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:26:43.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-02-05T06:26:43.933 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:d1337354:::obj10:head# 2024-02-05T06:26:44.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:44.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-02-05T06:26:44.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T06:26:44.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-02-05T06:26:44.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:44.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:44.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:26:44.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:44.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:26:44.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-02-05T06:26:45.411 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:eeae2a94:::obj11:head# 2024-02-05T06:26:46.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:46.196 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-02-05T06:26:46.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T06:26:46.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-02-05T06:26:46.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:46.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:46.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:26:46.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:46.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:26:46.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-02-05T06:26:47.038 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:af5f95cb:::obj12:head# 2024-02-05T06:26:47.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:47.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-02-05T06:26:47.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T06:26:47.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-02-05T06:26:47.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:47.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:47.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:26:47.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:47.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:26:47.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-02-05T06:26:48.568 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:a61ad63e:::obj13:head# 2024-02-05T06:26:49.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:49.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-02-05T06:26:49.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T06:26:49.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-02-05T06:26:49.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:49.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:49.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:26:49.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:49.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:26:49.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-02-05T06:26:49.786 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:e0a44829:::obj14:head# 2024-02-05T06:26:50.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:50.315 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-02-05T06:26:50.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T06:26:50.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-02-05T06:26:50.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:50.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:50.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:26:50.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:50.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:26:50.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-02-05T06:26:51.032 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:ab946124:::obj15:head# 2024-02-05T06:26:51.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:51.565 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-02-05T06:26:51.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T06:26:51.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-02-05T06:26:51.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:51.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:51.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:26:51.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:51.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:26:51.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-02-05T06:26:52.206 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:461f8b5e:::obj16:head# 2024-02-05T06:26:52.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:52.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-02-05T06:26:52.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T06:26:52.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-02-05T06:26:52.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:52.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:52.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:26:52.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:52.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:26:52.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-02-05T06:26:53.501 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:24fc6e92:::obj17:head# 2024-02-05T06:26:54.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:54.037 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-02-05T06:26:54.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T06:26:54.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-02-05T06:26:54.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:54.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:54.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:26:54.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:54.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:26:54.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-02-05T06:26:54.719 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:b9836a99:::obj18:head# 2024-02-05T06:26:55.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:55.257 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-02-05T06:26:55.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T06:26:55.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-02-05T06:26:55.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:55.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:55.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:26:55.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:55.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:26:55.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-02-05T06:26:55.937 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:00b31b6c:::obj19:head# 2024-02-05T06:26:56.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T06:26:56.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-02-05T06:26:56.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T06:26:56.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-02-05T06:26:56.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:26:56.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:26:56.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:26:56.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:26:56.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:26:56.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-02-05T06:26:57.273 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:24b5611b:::obj20:head# 2024-02-05T06:26:57.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:26:57.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:26:57.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:26:57.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:26:57.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:26:57.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:26:57.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:26:57.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:26:57.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:26:57.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:26:57.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:26:57.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:26:57.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:26:57.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:26:57.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:26:57.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:26:57.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:26:57.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:26:57.813 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:26:57.814 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:26:57.814 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:26:57.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:26:57.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:26:57.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:26:57.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:26:57.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:26:57.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:26:57.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:26:57.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:26:57.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:26:57.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:26:57.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T06:26:57.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:26:57.815 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:26:57.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:26:57.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:26:57.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:26:57.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:26:57.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:26:57.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:26:57.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:26:57.850 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:26:57.856+0000 7f4cc6212640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:26:57.853 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:26:57.856+0000 7f4cc6212640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:26:57.854 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:26:57.860+0000 7f4cc6212640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:26:58.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:26:58.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:26:58.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:26:58.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:26:58.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:26:58.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:26:58.139 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:26:58.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:26:58.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:58.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:58.399 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:26:58.404+0000 7f4cc6212640 -1 Falling back to public interface 2024-02-05T06:26:58.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:59.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:59.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:26:59.513 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:26:59.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:26:59.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:59.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:59.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:00.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:00.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:00.816 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:27:00.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:27:00.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:00.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:27:00.943 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:00.948+0000 7f4cc6212640 -1 osd.1 18 log_to_monitors true 2024-02-05T06:27:01.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:02.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:02.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:02.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:27:02.222 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:27:02.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:02.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:27:02.530 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [10,18) [v2:127.0.0.1:6802/4263508797,v1:127.0.0.1:6803/4263508797] [v2:127.0.0.1:6804/4263508797,v1:127.0.0.1:6805/4263508797] exists,up 15d0d83a-e44a-49a4-9cfe-b1413755c6e6 2024-02-05T06:27:02.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:27:02.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:27:02.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:27:02.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:27:02.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:27:02.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:27:02.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:27:02.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:27:02.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:27:02.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:27:02.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:27:02.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:27:02.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:27:02.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:27:02.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:27:02.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:27:02.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:27:02.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:27:02.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:27:02.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:27:02.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:27:02.534 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:27:02.534 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:02.534 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:27:02.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:27:02.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:27:02.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:27:02.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:27:02.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:27:02.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:27:02.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:27:02.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:27:02.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:27:02.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:27:02.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T06:27:02.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:27:02.537 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:27:02.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:27:02.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:27:02.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:27:02.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:27:02.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:27:02.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:27:02.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:27:02.567 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:02.572+0000 7ff6c08ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:27:02.598 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:02.604+0000 7ff6c08ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:27:02.600 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:02.604+0000 7ff6c08ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:27:02.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:27:02.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:27:02.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:27:02.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:27:02.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:27:02.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:02.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:27:02.852 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:27:02.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:02.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:03.144 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:03.148+0000 7ff6c08ba640 -1 Falling back to public interface 2024-02-05T06:27:03.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:04.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:04.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:04.165 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:27:04.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:27:04.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:04.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:04.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:05.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:05.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:05.518 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:27:05.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:27:05.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:05.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:05.653 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:05.656+0000 7ff6c08ba640 -1 osd.0 16 log_to_monitors true 2024-02-05T06:27:05.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:06.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:06.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:06.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:27:06.858 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:27:06.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:06.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:07.189 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6810/3039127878,v1:127.0.0.1:6811/3039127878] [v2:127.0.0.1:6812/3039127878,v1:127.0.0.1:6813/3039127878] exists,up 0e051bd0-257b-438f-b9ca-f0ebe6859f71 2024-02-05T06:27:07.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:27:07.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:27:07.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:27:07.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-02-05T06:27:07.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:27:07.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:27:07.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:27:07.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:27:07.190 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:27:07.191 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:27:07.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:27:07.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:27:07.191 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:27:07.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:27:07.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:27:07.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:27:07.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:27:07.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:27:07.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:27:07.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:27:07.609 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:27:07.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:27:07.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:27:07.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:27:07.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-02-05T06:27:07.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-02-05T06:27:07.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-02-05T06:27:07.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:27:07.772 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:27:07.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313219 2024-02-05T06:27:07.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313219 2024-02-05T06:27:07.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-90194313219' 2024-02-05T06:27:07.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:27:07.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-02-05T06:27:07.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:27:07.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:27:07.915 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-02-05T06:27:07.915 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:27:07.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-02-05T06:27:07.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-02-05T06:27:07.916 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 103079215106 2024-02-05T06:27:07.917 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:27:08.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 103079215106 2024-02-05T06:27:08.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:27:09.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:27:09.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:27:09.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-02-05T06:27:09.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:27:09.526 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-90194313219 2024-02-05T06:27:09.527 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:27:09.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:27:09.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-90194313219 2024-02-05T06:27:09.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:27:09.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313219 2024-02-05T06:27:09.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 90194313219' 2024-02-05T06:27:09.532 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 90194313219 2024-02-05T06:27:09.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:27:09.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313219 -lt 90194313219 2024-02-05T06:27:09.869 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:27:09.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:27:09.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:27:10.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:27:10.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:27:10.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:27:10.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:27:10.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:27:10.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:27:10.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:27:10.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:27:10.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:27:10.577 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:27:10.577 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:27:10.577 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:27:10.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:27:10.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:27:10.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:27:10.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-02-05T06:27:10.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-02-05T06:27:10.963 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-02-05T06:27:10.963 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:27:10.963 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:27:10.963 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:27:10.963 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:27:11.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T06:26:17.145989+0000 2024-02-05T06:27:11.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-02-05T06:27:11.572 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 1.0 on osd.1 to repair 2024-02-05T06:27:11.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-02-05T06:26:17.145989+0000 2024-02-05T06:27:11.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:27:11.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:26:17.145989+0000 2024-02-05T06:27:11.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:27:11.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:27:11.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:27:11.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:27:11.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:27:11.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:27:11.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:27:11.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:27:11.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:26:17.145989+0000 '>' 2024-02-05T06:26:17.145989+0000 2024-02-05T06:27:11.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:27:12.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:27:12.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:27:12.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:27:12.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:27:12.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:27:12.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:27:12.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:27:13.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:26:17.145989+0000 '>' 2024-02-05T06:26:17.145989+0000 2024-02-05T06:27:13.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:27:14.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:27:14.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:27:14.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:27:14.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:27:14.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:27:14.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:27:14.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:27:14.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:26:17.145989+0000 '>' 2024-02-05T06:26:17.145989+0000 2024-02-05T06:27:14.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:27:15.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:27:15.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:27:15.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:27:15.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:27:15.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:27:15.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:27:15.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:27:15.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:26:17.145989+0000 '>' 2024-02-05T06:26:17.145989+0000 2024-02-05T06:27:15.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:27:16.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:27:16.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:27:16.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:27:16.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:27:16.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:27:16.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:27:16.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:27:17.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:27:11.996111+0000 '>' 2024-02-05T06:26:17.145989+0000 2024-02-05T06:27:17.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:27:17.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-02-05T06:27:17.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:27:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:27:17.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:27:17.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:27:17.158 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:27:17.158 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:27:17.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:27:17.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:27:17.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:27:17.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:27:17.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:27:17.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:27:17.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:27:17.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:27:17.270 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:27:17.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:27:17.583 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:27:17.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:27:17.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:27:17.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:27:17.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215109 2024-02-05T06:27:17.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215109 2024-02-05T06:27:17.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109' 2024-02-05T06:27:17.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:27:17.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:27:17.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313222 2024-02-05T06:27:17.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313222 2024-02-05T06:27:17.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109 1-90194313222' 2024-02-05T06:27:17.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:27:17.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215109 2024-02-05T06:27:17.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:27:17.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:27:17.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215109 2024-02-05T06:27:17.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:27:17.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215109 2024-02-05T06:27:17.913 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 103079215109 2024-02-05T06:27:17.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215109' 2024-02-05T06:27:17.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:27:18.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215108 -lt 103079215109 2024-02-05T06:27:18.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:27:19.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:27:19.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:27:19.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215109 2024-02-05T06:27:19.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:27:19.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-90194313222 2024-02-05T06:27:19.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:27:19.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:27:19.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-90194313222 2024-02-05T06:27:19.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:27:19.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313222 2024-02-05T06:27:19.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 90194313222' 2024-02-05T06:27:19.533 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 90194313222 2024-02-05T06:27:19.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:27:19.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313222 -lt 90194313222 2024-02-05T06:27:19.872 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:27:19.872 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:27:19.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:27:20.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:27:20.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:27:20.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:27:20.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:27:20.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:27:20.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:27:20.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:27:20.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:27:20.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:27:20.588 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:27:20.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:27:20.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:27:20.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:27:20.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:27:20.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:27:20.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-02-05T06:27:21.307 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T06:27:21.307 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-02-05T06:27:13.030338+0000 16'30 25:120 [1,0] 1 [1,0] 1 16'30 2024-02-05T06:27:11.996111+0000 16'30 2024-02-05T06:27:11.996111+0000 0 1 periodic scrub scheduled @ 2024-02-06T06:27:11.996111+0000 20 0 2024-02-05T06:27:21.307 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T06:27:21.307 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T06:27:21.307 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:27:21.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-02-05T06:27:21.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:27:21.323 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:27:21.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:27:21.613 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:27:21.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:27:21.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:27:21.613 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:27:21.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215111 2024-02-05T06:27:21.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215111 2024-02-05T06:27:21.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215111' 2024-02-05T06:27:21.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:27:21.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:27:21.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313224 2024-02-05T06:27:21.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313224 2024-02-05T06:27:21.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215111 1-90194313224' 2024-02-05T06:27:21.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:27:21.934 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215111 2024-02-05T06:27:21.934 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:27:21.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:27:21.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215111 2024-02-05T06:27:21.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:27:21.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215111 2024-02-05T06:27:21.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215111' 2024-02-05T06:27:21.937 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 103079215111 2024-02-05T06:27:21.937 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:27:22.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215111 2024-02-05T06:27:22.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:27:23.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:27:23.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:27:23.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215111 -lt 103079215111 2024-02-05T06:27:23.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:27:23.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-90194313224 2024-02-05T06:27:23.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:27:23.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:27:23.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-90194313224 2024-02-05T06:27:23.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:27:23.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313224 2024-02-05T06:27:23.523 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 90194313224 2024-02-05T06:27:23.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 90194313224' 2024-02-05T06:27:23.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:27:23.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313224 -lt 90194313224 2024-02-05T06:27:23.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-02-05T06:27:23.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-02-05T06:27:24.015 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:27:24.015 INFO:tasks.workunit.client.0.smithi148.stdout: "num_bytes": 210, 2024-02-05T06:27:24.015 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects": 30, 2024-02-05T06:27:24.015 INFO:tasks.workunit.client.0.smithi148.stdout: "num_object_clones": 0, 2024-02-05T06:27:24.015 INFO:tasks.workunit.client.0.smithi148.stdout: "num_object_copies": 60, 2024-02-05T06:27:24.015 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_missing_on_primary": 0, 2024-02-05T06:27:24.015 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_missing": 0, 2024-02-05T06:27:24.015 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_degraded": 0, 2024-02-05T06:27:24.016 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_misplaced": 0, 2024-02-05T06:27:24.016 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_unfound": 0, 2024-02-05T06:27:24.016 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_dirty": 30, 2024-02-05T06:27:24.016 INFO:tasks.workunit.client.0.smithi148.stdout: "num_whiteouts": 0, 2024-02-05T06:27:24.016 INFO:tasks.workunit.client.0.smithi148.stdout: "num_read": 0, 2024-02-05T06:27:24.016 INFO:tasks.workunit.client.0.smithi148.stdout: "num_read_kb": 0, 2024-02-05T06:27:24.016 INFO:tasks.workunit.client.0.smithi148.stdout: "num_write": 30, 2024-02-05T06:27:24.016 INFO:tasks.workunit.client.0.smithi148.stdout: "num_write_kb": 30, 2024-02-05T06:27:24.016 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrub_errors": 0, 2024-02-05T06:27:24.016 INFO:tasks.workunit.client.0.smithi148.stdout: "num_shallow_scrub_errors": 0, 2024-02-05T06:27:24.016 INFO:tasks.workunit.client.0.smithi148.stdout: "num_deep_scrub_errors": 0, 2024-02-05T06:27:24.016 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_recovered": 20, 2024-02-05T06:27:24.016 INFO:tasks.workunit.client.0.smithi148.stdout: "num_bytes_recovered": 140, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_keys_recovered": 0, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_omap": 0, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_hit_set_archive": 0, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_bytes_hit_set_archive": 0, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_flush": 0, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_flush_kb": 0, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_evict": 0, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_evict_kb": 0, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_promote": 0, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_flush_mode_high": 0, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_flush_mode_low": 0, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_evict_mode_some": 0, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_evict_mode_full": 0, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_pinned": 0, 2024-02-05T06:27:24.017 INFO:tasks.workunit.client.0.smithi148.stdout: "num_legacy_snapsets": 0, 2024-02-05T06:27:24.018 INFO:tasks.workunit.client.0.smithi148.stdout: "num_large_omap_objects": 0, 2024-02-05T06:27:24.018 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_manifest": 0, 2024-02-05T06:27:24.018 INFO:tasks.workunit.client.0.smithi148.stdout: "num_omap_bytes": 0, 2024-02-05T06:27:24.018 INFO:tasks.workunit.client.0.smithi148.stdout: "num_omap_keys": 0, 2024-02-05T06:27:24.018 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_repaired": 20 2024-02-05T06:27:24.018 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:27:24.018 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-02-05T06:27:24.018 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-02-05T06:27:24.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-02-05T06:27:24.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-02-05T06:27:24.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-02-05T06:27:24.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-02-05T06:27:24.442 INFO:tasks.workunit.client.0.smithi148.stderr:dumped all 2024-02-05T06:27:24.459 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:27:24.459 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:27:24.460 INFO:tasks.workunit.client.0.smithi148.stdout: "up_from": 21, 2024-02-05T06:27:24.460 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 90194313224, 2024-02-05T06:27:24.460 INFO:tasks.workunit.client.0.smithi148.stdout: "num_pgs": 1, 2024-02-05T06:27:24.460 INFO:tasks.workunit.client.0.smithi148.stdout: "num_osds": 1, 2024-02-05T06:27:24.460 INFO:tasks.workunit.client.0.smithi148.stdout: "num_per_pool_osds": 1, 2024-02-05T06:27:24.460 INFO:tasks.workunit.client.0.smithi148.stdout: "num_per_pool_omap_osds": 1, 2024-02-05T06:27:24.460 INFO:tasks.workunit.client.0.smithi148.stdout: "kb": 104857600, 2024-02-05T06:27:24.460 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used": 28560, 2024-02-05T06:27:24.460 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_data": 328, 2024-02-05T06:27:24.460 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_omap": 4, 2024-02-05T06:27:24.460 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_meta": 28219, 2024-02-05T06:27:24.460 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_avail": 104829040, 2024-02-05T06:27:24.460 INFO:tasks.workunit.client.0.smithi148.stdout: "statfs": { 2024-02-05T06:27:24.460 INFO:tasks.workunit.client.0.smithi148.stdout: "total": 107374182400, 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: "available": 107344936960, 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: "internally_reserved": 0, 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: "allocated": 335872, 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: "data_stored": 55848, 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed": 0, 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed_allocated": 0, 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed_original": 0, 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_allocated": 4398, 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: "internal_metadata": 28896978 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: "hb_peers": [ 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: 0 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: "snap_trim_queue_len": 0, 2024-02-05T06:27:24.461 INFO:tasks.workunit.client.0.smithi148.stdout: "num_snap_trimming": 0, 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: "num_shards_repaired": 10, 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: "op_queue_age_hist": { 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: "histogram": [], 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: "upper_bound": 1 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: "perf_stat": { 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: "commit_latency_ms": 0, 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: "apply_latency_ms": 0, 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: "commit_latency_ns": 0, 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: "apply_latency_ns": 0 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: "alerts": [], 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: "network_ping_times": [ 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:27:24.462 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: "interfaces": [ 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: "interface": "back", 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: "average": { 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: "min": { 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:24.463 INFO:tasks.workunit.client.0.smithi148.stdout: "max": { 2024-02-05T06:27:24.464 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:27:24.464 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:27:24.464 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:27:24.464 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:24.464 INFO:tasks.workunit.client.0.smithi148.stdout: "last": 0.388 2024-02-05T06:27:24.464 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:27:24.464 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:27:24.464 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:27:24.464 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:27:24.464 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:27:24.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-02-05T06:27:24.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-02-05T06:27:24.740 INFO:tasks.workunit.client.0.smithi148.stderr:dumped all 2024-02-05T06:27:24.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-02-05T06:27:24.757 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-02-05T06:27:24.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-02-05T06:27:24.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-02-05T06:27:24.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-02-05T06:27:25.053 INFO:tasks.workunit.client.0.smithi148.stderr:dumped all 2024-02-05T06:27:25.068 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:27:25.068 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:27:25.068 INFO:tasks.workunit.client.0.smithi148.stdout: "up_from": 24, 2024-02-05T06:27:25.068 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 103079215111, 2024-02-05T06:27:25.068 INFO:tasks.workunit.client.0.smithi148.stdout: "num_pgs": 1, 2024-02-05T06:27:25.068 INFO:tasks.workunit.client.0.smithi148.stdout: "num_osds": 1, 2024-02-05T06:27:25.069 INFO:tasks.workunit.client.0.smithi148.stdout: "num_per_pool_osds": 1, 2024-02-05T06:27:25.069 INFO:tasks.workunit.client.0.smithi148.stdout: "num_per_pool_omap_osds": 1, 2024-02-05T06:27:25.069 INFO:tasks.workunit.client.0.smithi148.stdout: "kb": 104857600, 2024-02-05T06:27:25.069 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used": 28560, 2024-02-05T06:27:25.069 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_data": 328, 2024-02-05T06:27:25.069 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_omap": 4, 2024-02-05T06:27:25.069 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_meta": 28219, 2024-02-05T06:27:25.069 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_avail": 104829040, 2024-02-05T06:27:25.069 INFO:tasks.workunit.client.0.smithi148.stdout: "statfs": { 2024-02-05T06:27:25.069 INFO:tasks.workunit.client.0.smithi148.stdout: "total": 107374182400, 2024-02-05T06:27:25.069 INFO:tasks.workunit.client.0.smithi148.stdout: "available": 107344936960, 2024-02-05T06:27:25.069 INFO:tasks.workunit.client.0.smithi148.stdout: "internally_reserved": 0, 2024-02-05T06:27:25.069 INFO:tasks.workunit.client.0.smithi148.stdout: "allocated": 335872, 2024-02-05T06:27:25.069 INFO:tasks.workunit.client.0.smithi148.stdout: "data_stored": 55848, 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed": 0, 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed_allocated": 0, 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed_original": 0, 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_allocated": 4401, 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: "internal_metadata": 28896975 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: "hb_peers": [ 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: "snap_trim_queue_len": 0, 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: "num_snap_trimming": 0, 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: "num_shards_repaired": 10, 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: "op_queue_age_hist": { 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: "histogram": [], 2024-02-05T06:27:25.070 INFO:tasks.workunit.client.0.smithi148.stdout: "upper_bound": 1 2024-02-05T06:27:25.071 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:25.071 INFO:tasks.workunit.client.0.smithi148.stdout: "perf_stat": { 2024-02-05T06:27:25.071 INFO:tasks.workunit.client.0.smithi148.stdout: "commit_latency_ms": 0, 2024-02-05T06:27:25.071 INFO:tasks.workunit.client.0.smithi148.stdout: "apply_latency_ms": 0, 2024-02-05T06:27:25.071 INFO:tasks.workunit.client.0.smithi148.stdout: "commit_latency_ns": 0, 2024-02-05T06:27:25.071 INFO:tasks.workunit.client.0.smithi148.stdout: "apply_latency_ns": 0 2024-02-05T06:27:25.071 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:25.071 INFO:tasks.workunit.client.0.smithi148.stdout: "alerts": [], 2024-02-05T06:27:25.071 INFO:tasks.workunit.client.0.smithi148.stdout: "network_ping_times": [ 2024-02-05T06:27:25.071 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:27:25.071 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:27:25.071 INFO:tasks.workunit.client.0.smithi148.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T06:27:25.071 INFO:tasks.workunit.client.0.smithi148.stdout: "interfaces": [ 2024-02-05T06:27:25.071 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:27:25.072 INFO:tasks.workunit.client.0.smithi148.stdout: "interface": "back", 2024-02-05T06:27:25.072 INFO:tasks.workunit.client.0.smithi148.stdout: "average": { 2024-02-05T06:27:25.072 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:27:25.072 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:27:25.072 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:27:25.072 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:25.072 INFO:tasks.workunit.client.0.smithi148.stdout: "min": { 2024-02-05T06:27:25.072 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:27:25.072 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:27:25.072 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:27:25.072 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:25.072 INFO:tasks.workunit.client.0.smithi148.stdout: "max": { 2024-02-05T06:27:25.072 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:27:25.072 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:27:25.073 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:27:25.073 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:25.073 INFO:tasks.workunit.client.0.smithi148.stdout: "last": 0.42 2024-02-05T06:27:25.073 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:27:25.073 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:27:25.073 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:27:25.073 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:27:25.073 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:27:25.073 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-02-05T06:27:25.074 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-02-05T06:27:25.335 INFO:tasks.workunit.client.0.smithi148.stderr:dumped all 2024-02-05T06:27:25.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-02-05T06:27:25.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-02-05T06:27:25.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-02-05T06:27:25.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-02-05T06:27:25.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-02-05T06:27:25.617 INFO:tasks.workunit.client.0.smithi148.stderr:dumped all 2024-02-05T06:27:25.632 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:27:25.632 INFO:tasks.workunit.client.0.smithi148.stdout: "up_from": 0, 2024-02-05T06:27:25.632 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 0, 2024-02-05T06:27:25.632 INFO:tasks.workunit.client.0.smithi148.stdout: "num_pgs": 2, 2024-02-05T06:27:25.632 INFO:tasks.workunit.client.0.smithi148.stdout: "num_osds": 2, 2024-02-05T06:27:25.632 INFO:tasks.workunit.client.0.smithi148.stdout: "num_per_pool_osds": 2, 2024-02-05T06:27:25.632 INFO:tasks.workunit.client.0.smithi148.stdout: "num_per_pool_omap_osds": 2, 2024-02-05T06:27:25.632 INFO:tasks.workunit.client.0.smithi148.stdout: "kb": 209715200, 2024-02-05T06:27:25.632 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used": 57120, 2024-02-05T06:27:25.632 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_data": 656, 2024-02-05T06:27:25.632 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_omap": 8, 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_meta": 56439, 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_avail": 209658080, 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "statfs": { 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "total": 214748364800, 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "available": 214689873920, 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "internally_reserved": 0, 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "allocated": 671744, 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "data_stored": 111696, 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed": 0, 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed_allocated": 0, 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed_original": 0, 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_allocated": 8799, 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "internal_metadata": 57793953 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "hb_peers": [], 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "snap_trim_queue_len": 0, 2024-02-05T06:27:25.633 INFO:tasks.workunit.client.0.smithi148.stdout: "num_snap_trimming": 0, 2024-02-05T06:27:25.634 INFO:tasks.workunit.client.0.smithi148.stdout: "num_shards_repaired": 20, 2024-02-05T06:27:25.634 INFO:tasks.workunit.client.0.smithi148.stdout: "op_queue_age_hist": { 2024-02-05T06:27:25.634 INFO:tasks.workunit.client.0.smithi148.stdout: "histogram": [], 2024-02-05T06:27:25.634 INFO:tasks.workunit.client.0.smithi148.stdout: "upper_bound": 1 2024-02-05T06:27:25.634 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:25.634 INFO:tasks.workunit.client.0.smithi148.stdout: "perf_stat": { 2024-02-05T06:27:25.634 INFO:tasks.workunit.client.0.smithi148.stdout: "commit_latency_ms": 0, 2024-02-05T06:27:25.634 INFO:tasks.workunit.client.0.smithi148.stdout: "apply_latency_ms": 0, 2024-02-05T06:27:25.634 INFO:tasks.workunit.client.0.smithi148.stdout: "commit_latency_ns": 0, 2024-02-05T06:27:25.634 INFO:tasks.workunit.client.0.smithi148.stdout: "apply_latency_ns": 0 2024-02-05T06:27:25.634 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:27:25.634 INFO:tasks.workunit.client.0.smithi148.stdout: "alerts": [], 2024-02-05T06:27:25.634 INFO:tasks.workunit.client.0.smithi148.stdout: "network_ping_times": [] 2024-02-05T06:27:25.634 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:27:25.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-02-05T06:27:25.635 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-02-05T06:27:25.900 INFO:tasks.workunit.client.0.smithi148.stderr:dumped all 2024-02-05T06:27:25.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-02-05T06:27:25.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-02-05T06:27:25.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:27:25.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:27:25.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:27:25.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:27:25.918 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:27:25.918 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:27:25.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:27:25.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:27:25.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:27:26.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:27:26.229 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:27:26.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:27:26.230 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:27:26.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:27:26.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:27:26.233 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:27:26.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:27:26.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:27:26.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:27:26.235 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:27:26.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:27:26.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:27:26.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:27:26.256 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:27:26.256 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:26.256 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:27:26.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:27:26.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:27:26.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:27:26.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:27:26.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:27:26.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:27:26.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:27:26.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:27:26.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:27:26.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:27:26.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:27:26.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:27:26.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:27:26.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:27:26.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:27:26.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:27:26.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:27:26.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:27:26.265 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:27:26.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:27:26.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:27:26.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:27:26.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:27:26.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:27:26.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:27:26.269 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:27:26.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:27:26.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:27:26.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:27:26.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:27:26.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:26.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:27:26.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:27:26.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:27:26.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:27:26.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:27:26.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:27:26.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:26.277 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:27:26.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T06:27:26.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:27:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:27:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:27:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:27:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-02-05T06:27:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-02-05T06:27:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-02-05T06:27:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-02-05T06:27:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-02-05T06:27:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-02-05T06:27:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-02-05T06:27:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-02-05T06:27:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:27:26.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:27:26.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:27:26.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:27:26.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:27:26.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T06:27:26.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:27:26.681 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:27:26.681 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:27:26.681 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:27:26.681 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:26.682 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:27:26.682 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:27:26.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T06:27:26.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:27:26.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:27:26.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:27:26.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:27:26.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:27:26.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:27:26.899 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:27:26.899 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:27:26.899 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:27:26.900 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:27:26.900 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:26.900 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:27:26.900 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:27:26.901 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:27:26.901 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T06:27:27.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:27:27.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:27:27.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:27:27.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:27:27.018 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:27:27.018 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:27:27.018 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:27:27.018 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:27:27.018 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:27:27.018 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:27.018 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:27:27.018 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:27:27.019 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:27:27.019 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T06:27:27.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-02-05T06:27:27.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:27:27.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:27:27.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:27:27.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:27:27.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:27:27.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:27:28.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:27:28.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:27:28.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:27:28.016 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:27:28.016 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:28.016 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:27:28.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:27:28.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:27:28.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:27:28.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T06:27:28.055 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-02-05T06:27:28.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-02-05T06:27:28.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:27:28.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:27:28.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:27:28.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:27:28.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:27:28.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:27:28.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:27:28.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:27:28.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:27:28.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:27:28.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:27:28.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:27:28.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:27:28.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:27:28.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:27:28.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:27:28.060 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:27:28.060 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:27:28.060 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:27:28.060 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:27:28.060 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:28.061 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:27:28.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:27:28.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:27:28.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:27:28.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:27:28.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:27:28.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:27:28.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:27:28.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:27:28.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:27:28.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:27:28.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:27:28.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:27:28.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T06:27:28.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:27:28.063 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:27:28.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=63609b06-9d96-4822-b492-6b09cd860fc1 2024-02-05T06:27:28.064 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 63609b06-9d96-4822-b492-6b09cd860fc1 2024-02-05T06:27:28.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 63609b06-9d96-4822-b492-6b09cd860fc1' 2024-02-05T06:27:28.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:27:28.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQf8Bl8T42BRAAsaiEXKsC8o4nUM8rNRLI6w== 2024-02-05T06:27:28.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQf8Bl8T42BRAAsaiEXKsC8o4nUM8rNRLI6w=="}' 2024-02-05T06:27:28.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 63609b06-9d96-4822-b492-6b09cd860fc1 -i td/osd-scrub-repair/0/new.json 2024-02-05T06:27:28.395 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:27:28.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:27:28.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDQf8Bl8T42BRAAsaiEXKsC8o4nUM8rNRLI6w== --osd-uuid 63609b06-9d96-4822-b492-6b09cd860fc1 2024-02-05T06:27:28.448 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:28.451+0000 7f1e28873640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:27:28.479 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:28.483+0000 7f1e28873640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:27:28.480 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:28.483+0000 7f1e28873640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:27:28.481 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:28.483+0000 7f1e28873640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:27:28.481 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:28.483+0000 7f1e28873640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:27:28.481 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:28.483+0000 7f1e28873640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:27:28.571 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:28.575+0000 7f1e28873640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:27:34.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:27:34.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:27:34.659 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:27:34.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:27:34.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:27:35.048 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:27:35.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:27:35.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:27:35.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:27:35.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:27:35.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:27:35.083 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:35.087+0000 7f8d00c41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:27:35.121 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:35.123+0000 7f8d00c41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:27:35.122 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:35.123+0000 7f8d00c41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:27:35.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:27:35.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:27:35.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:27:35.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:27:35.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:27:35.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:35.401 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:27:35.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:27:35.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:35.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:35.676 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:35.679+0000 7f8d00c41640 -1 Falling back to public interface 2024-02-05T06:27:35.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:36.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:36.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:36.742 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:27:36.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:27:36.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:36.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:37.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:38.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:38.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:38.047 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:27:38.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:27:38.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:38.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:38.214 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:27:38.215+0000 7f8d00c41640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:27:38.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:39.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:39.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:39.395 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:27:39.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:27:39.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:39.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:39.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:40.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:40.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:40.715 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:27:40.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:27:40.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:40.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:41.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:42.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:42.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:42.039 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:27:42.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:27:42.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:42.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:42.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:43.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:43.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:43.359 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:27:43.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:27:43.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:43.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:43.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:44.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:44.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:44.682 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:27:44.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:27:44.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:44.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:45.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:46.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:46.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:46.023 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:27:46.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:27:46.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:46.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:46.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:47.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:47.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:47.332 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:27:47.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:27:47.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:47.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:47.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:48.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:48.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:48.640 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:27:48.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:27:48.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:48.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:49.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:50.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:50.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:50.019 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:27:50.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:27:50.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:50.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:50.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:51.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:51.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:51.336 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:27:51.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:27:51.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:51.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:51.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:52.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:52.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:52.859 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:27:52.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:27:52.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:52.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:53.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:54.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:54.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:54.475 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:27:54.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:27:54.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:54.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:54.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:55.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:55.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:55.959 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:27:55.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:27:55.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:55.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:56.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:57.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:57.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:57.494 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:27:57.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:27:57.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:57.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:57.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:27:58.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:27:58.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:27:58.872 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:27:58.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:27:58.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:27:58.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:27:59.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:00.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:00.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:00.727 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:28:00.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:28:00.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:00.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:28:01.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:02.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:02.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:02.124 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:28:02.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:28:02.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:02.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:28:02.447 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1616081413,v1:127.0.0.1:6803/1616081413] [v2:127.0.0.1:6804/1616081413,v1:127.0.0.1:6805/1616081413] exists,up 63609b06-9d96-4822-b492-6b09cd860fc1 2024-02-05T06:28:02.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:28:02.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:28:02.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:28:02.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:28:02.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:28:02.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:28:02.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:28:02.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:28:02.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:28:02.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:28:02.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:28:02.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:28:02.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:28:02.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:28:02.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:28:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:28:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:28:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:28:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:28:02.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:28:02.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:28:02.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:28:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:28:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:28:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:28:02.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:28:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:28:02.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:28:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:28:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:28:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:28:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:28:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:28:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:28:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:28:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:28:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:28:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T06:28:02.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:28:02.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:28:02.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fde606da-da07-46a8-a1f9-b3fbf3d0e18d 2024-02-05T06:28:02.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fde606da-da07-46a8-a1f9-b3fbf3d0e18d' 2024-02-05T06:28:02.454 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 fde606da-da07-46a8-a1f9-b3fbf3d0e18d 2024-02-05T06:28:02.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:28:02.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDyf8Ble5y2HBAAUb5XFI0jAhAXvyK9LjzRmQ== 2024-02-05T06:28:02.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDyf8Ble5y2HBAAUb5XFI0jAhAXvyK9LjzRmQ=="}' 2024-02-05T06:28:02.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fde606da-da07-46a8-a1f9-b3fbf3d0e18d -i td/osd-scrub-repair/1/new.json 2024-02-05T06:28:02.837 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:28:02.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T06:28:02.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDyf8Ble5y2HBAAUb5XFI0jAhAXvyK9LjzRmQ== --osd-uuid fde606da-da07-46a8-a1f9-b3fbf3d0e18d 2024-02-05T06:28:02.889 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:02.890+0000 7f873ce95640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:02.931 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:02.934+0000 7f873ce95640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:02.932 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:02.934+0000 7f873ce95640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:02.933 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:02.934+0000 7f873ce95640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:28:02.933 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:02.934+0000 7f873ce95640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:28:02.933 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:02.934+0000 7f873ce95640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:28:03.002 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:03.002+0000 7f873ce95640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T06:28:09.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T06:28:09.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:28:09.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:28:09.448 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:28:09.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:28:09.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:28:09.836 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:28:09.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:28:09.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:28:09.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:28:09.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:28:09.867 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:09.866+0000 7ff90636a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:09.894 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:09.894+0000 7ff90636a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:09.895 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:09.894+0000 7ff90636a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:10.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:28:10.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:28:10.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:28:10.153 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:28:10.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:28:10.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:28:10.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:10.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:28:10.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:10.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:10.446 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:10.446+0000 7ff90636a640 -1 Falling back to public interface 2024-02-05T06:28:10.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:11.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:11.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:11.494 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:28:11.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:28:11.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:11.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:11.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:12.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:12.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:12.794 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:28:12.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:28:12.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:12.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:13.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:13.216 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:13.218+0000 7ff90636a640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:28:14.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:14.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:14.118 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:28:14.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:28:14.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:14.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:14.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:15.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:15.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:15.497 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:28:15.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:28:15.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:15.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:15.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:16.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:16.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:16.817 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:28:16.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:28:16.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:16.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:17.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:18.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:18.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:18.182 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:28:18.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:28:18.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:18.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:18.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:19.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:19.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:19.501 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:28:19.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:28:19.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:19.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:19.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:20.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:20.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:20.820 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:28:20.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:28:20.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:20.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:21.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:22.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:22.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:22.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:28:22.159 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:28:22.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:22.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:22.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:23.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:23.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:23.471 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:28:23.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:28:23.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:23.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:23.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:24.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:24.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:24.789 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:28:24.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:28:24.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:24.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:25.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:26.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:26.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:26.147 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:28:26.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:28:26.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:26.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:26.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:27.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:27.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:27.884 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:28:27.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:28:27.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:27.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:28.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:29.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:29.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:29.662 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:28:29.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:28:29.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:29.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:30.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:31.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:31.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:31.507 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:28:31.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:28:31.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:31.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:32.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:33.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:33.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:33.122 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:28:33.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:28:33.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:33.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:33.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:34.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:34.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:34.846 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:28:34.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:28:34.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:34.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:35.285 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:36.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:36.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:36.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:28:36.288 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:28:36.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:36.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:36.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:37.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:37.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:37.894 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:28:37.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:28:37.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:37.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:28:38.206 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/142407934,v1:127.0.0.1:6811/142407934] [v2:127.0.0.1:6812/142407934,v1:127.0.0.1:6813/142407934] exists,up fde606da-da07-46a8-a1f9-b3fbf3d0e18d 2024-02-05T06:28:38.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:28:38.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:28:38.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:28:38.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:28:38.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:28:38.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:28:38.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:28:38.211 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:28:38.211 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:28:38.211 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:28:38.211 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:28:38.211 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:28:38.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:28:38.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:28:38.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:28:38.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:28:38.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:28:38.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:28:38.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:28:38.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:28:38.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:28:38.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:28:38.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:28:38.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T06:28:38.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:28:38.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:28:38.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=be761295-59af-470f-a9c6-c7d74e61add2 2024-02-05T06:28:38.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 be761295-59af-470f-a9c6-c7d74e61add2' 2024-02-05T06:28:38.214 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 be761295-59af-470f-a9c6-c7d74e61add2 2024-02-05T06:28:38.214 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:28:38.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAWgMBlaqtODhAA+oj/K5v2VN068NBrwGrIkw== 2024-02-05T06:28:38.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAWgMBlaqtODhAA+oj/K5v2VN068NBrwGrIkw=="}' 2024-02-05T06:28:38.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new be761295-59af-470f-a9c6-c7d74e61add2 -i td/osd-scrub-repair/2/new.json 2024-02-05T06:28:38.681 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:28:38.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T06:28:38.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAWgMBlaqtODhAA+oj/K5v2VN068NBrwGrIkw== --osd-uuid be761295-59af-470f-a9c6-c7d74e61add2 2024-02-05T06:28:38.741 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:38.746+0000 7fd699cba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:38.795 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:38.798+0000 7fd699cba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:38.796 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:38.802+0000 7fd699cba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:38.797 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:38.802+0000 7fd699cba640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:28:38.797 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:38.802+0000 7fd699cba640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:28:38.797 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:38.802+0000 7fd699cba640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:28:38.849 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:38.854+0000 7fd699cba640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T06:28:45.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T06:28:45.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:28:45.216 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T06:28:45.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T06:28:45.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:28:45.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T06:28:45.596 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T06:28:45.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:28:45.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:28:45.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:28:45.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:28:45.629 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:45.633+0000 7f9a105c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:45.657 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:45.661+0000 7f9a105c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:45.659 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:45.661+0000 7f9a105c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:45.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T06:28:45.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:28:45.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:28:45.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:28:45.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:28:45.924 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:28:45.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:45.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:28:45.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:45.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:28:46.206 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:46.209+0000 7f9a105c2640 -1 Falling back to public interface 2024-02-05T06:28:46.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:47.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:47.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:47.237 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:28:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:28:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:47.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:28:47.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:48.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:48.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:48.545 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:28:48.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:28:48.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:48.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:28:48.768 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:28:48.769+0000 7f9a105c2640 -1 osd.2 0 log_to_monitors true 2024-02-05T06:28:48.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:49.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:49.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:49.883 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:28:49.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:28:49.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:49.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:28:50.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:51.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:51.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:51.201 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:28:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:28:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:51.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:28:51.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:52.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:52.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:52.634 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:28:52.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:28:52.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:52.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:28:52.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:53.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:53.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:53.958 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:28:53.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:28:53.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:53.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:28:54.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:55.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:55.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:55.273 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:28:55.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:28:55.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:55.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:28:55.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:56.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:56.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:56.776 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:28:56.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:28:56.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:56.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:28:57.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:58.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:58.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:58.094 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:28:58.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:28:58.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:58.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:28:58.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:28:59.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:28:59.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:28:59.463 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:28:59.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:28:59.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:28:59.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:28:59.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:29:00.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:29:00.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:29:00.830 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:29:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:29:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:29:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:29:01.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:29:02.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:29:02.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:29:02.240 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:29:02.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:29:02.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:29:02.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:29:02.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:29:03.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:29:03.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:29:03.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:29:03.551 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:29:03.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:29:03.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:29:04.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:29:05.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:29:05.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:29:05.357 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:29:05.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:29:05.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:29:05.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:29:05.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:29:06.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:29:06.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:29:06.821 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:29:06.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:29:06.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:29:06.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:29:07.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:29:08.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:29:08.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:29:08.678 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:29:08.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:29:08.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:29:08.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:29:09.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:29:10.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:29:10.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:29:10.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:29:10.393 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:29:10.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:29:10.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:29:11.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:29:12.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:29:12.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:29:12.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:29:12.167 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:29:12.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:29:12.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:29:12.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:29:13.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:29:13.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:29:13.733 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:29:13.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:29:13.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:29:13.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:29:14.071 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3990677128,v1:127.0.0.1:6819/3990677128] [v2:127.0.0.1:6820/3990677128,v1:127.0.0.1:6821/3990677128] exists,up be761295-59af-470f-a9c6-c7d74e61add2 2024-02-05T06:29:14.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:29:14.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:29:14.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:29:14.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-02-05T06:29:14.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=testpool 2024-02-05T06:29:14.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T06:29:14.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-02-05T06:29:14.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T06:29:14.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-02-05T06:29:14.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-02-05T06:29:14.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-02-05T06:29:15.298 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'testpool' created 2024-02-05T06:29:15.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:29:16.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-02-05T06:29:16.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T06:29:16.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:29:16.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:29:16.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:29:16.448 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:29:16.449 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:29:16.449 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:29:16.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:29:16.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:29:16.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:29:16.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:29:16.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:29:16.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:29:16.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:29:16.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:29:16.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:29:16.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:29:16.816 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:29:16.816 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:29:16.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:29:16.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:29:16.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:29:17.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-02-05T06:29:17.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-02-05T06:29:17.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-02-05T06:29:17.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:29:17.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:29:17.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-02-05T06:29:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-02-05T06:29:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672971' 2024-02-05T06:29:17.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:29:17.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:29:17.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-02-05T06:29:17.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-02-05T06:29:17.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672971 2-64424509446' 2024-02-05T06:29:17.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:29:17.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-02-05T06:29:17.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:29:17.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:29:17.417 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-02-05T06:29:17.417 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:29:17.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-02-05T06:29:17.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-02-05T06:29:17.419 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836505 2024-02-05T06:29:17.419 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:29:17.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836505 2024-02-05T06:29:17.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:29:18.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:29:18.735 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:29:19.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-02-05T06:29:19.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:29:19.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-02-05T06:29:19.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:29:19.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:29:19.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-02-05T06:29:19.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:29:19.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-02-05T06:29:19.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-02-05T06:29:19.061 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672971 2024-02-05T06:29:19.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:29:19.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672971 2024-02-05T06:29:19.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:29:19.383 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-02-05T06:29:19.383 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:29:19.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:29:19.386 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-02-05T06:29:19.386 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:29:19.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-02-05T06:29:19.388 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509446 2024-02-05T06:29:19.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-02-05T06:29:19.389 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:29:19.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-02-05T06:29:19.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:29:19.732 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:29:19.732 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:29:20.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:29:20.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:29:20.113 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:29:20.113 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:29:20.113 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:29:20.113 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:29:20.113 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:29:20.113 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:29:20.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:29:20.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:29:20.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:29:20.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:29:20.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:29:20.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:29:20.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:29:20.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T06:29:20.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-02-05T06:29:20.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-02-05T06:29:20.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-02-05T06:29:20.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:20.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:20.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:20.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:21.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:21.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:21.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:21.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:21.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:21.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:21.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:21.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:22.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:22.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:22.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:22.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:22.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:22.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:22.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:22.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:22.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:22.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:23.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:23.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:23.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:23.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:23.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:23.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:23.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:23.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:24.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:24.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:24.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:24.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:24.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:24.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:24.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:24.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:25.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:25.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:25.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:25.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:25.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:25.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:25.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:25.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:26.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:26.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:26.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:26.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:26.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:26.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:26.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:26.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:27.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:27.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:27.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T06:29:27.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-02-05T06:29:27.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-02-05T06:29:27.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T06:29:27.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-02-05T06:29:27.620 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-02-05T06:29:27.620 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T06:29:27.620 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:29:27.621 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-02-05T06:29:27.621 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:29:27.942 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T06:29:27.943 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-02-05T06:29:27.943 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T06:29:28.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-02-05T06:29:28.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-02-05T06:29:28.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-02-05T06:29:28.275 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-02-05T06:29:28.275 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-02-05T06:29:28.275 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:29:28.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-02-05T06:29:28.600 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-02-05T06:29:28.600 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T06:29:28.600 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:29:28.600 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-02-05T06:29:28.600 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:29:28.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-02-05T06:29:28.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:29:28.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:29:28.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:29:28.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:29:28.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:29:28.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:29:29.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:29:29.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T06:29:29.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:29:29.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:29:29.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:29:29.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:29:29.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:29:29.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:29:29.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-02-05T06:29:29.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:29.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-02-05T06:29:29.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T06:29:29.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-02-05T06:29:29.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:29.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:29.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:29:29.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:29.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:29:29.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-02-05T06:29:30.946 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-02-05T06:29:31.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:31.483 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-02-05T06:29:31.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T06:29:31.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-02-05T06:29:31.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:31.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:31.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:29:31.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:31.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:29:31.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-02-05T06:29:32.726 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#1:104778fc:::obj2:head# 2024-02-05T06:29:33.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:33.275 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-02-05T06:29:33.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T06:29:33.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-02-05T06:29:33.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:33.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:33.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:29:33.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:33.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:29:33.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-02-05T06:29:34.229 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-02-05T06:29:34.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:34.763 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-02-05T06:29:34.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T06:29:34.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-02-05T06:29:34.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:34.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:34.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:29:34.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:34.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:29:34.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-02-05T06:29:35.641 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-02-05T06:29:36.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:36.176 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-02-05T06:29:36.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T06:29:36.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-02-05T06:29:36.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:36.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:36.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:29:36.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:36.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:29:36.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-02-05T06:29:36.976 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#1:c52c9666:::obj5:head# 2024-02-05T06:29:37.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:37.510 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-02-05T06:29:37.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T06:29:37.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-02-05T06:29:37.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:37.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:37.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:29:37.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:37.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:29:37.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-02-05T06:29:38.287 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#1:233a42c1:::obj6:head# 2024-02-05T06:29:38.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:38.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-02-05T06:29:38.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T06:29:38.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-02-05T06:29:38.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:38.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:38.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:29:38.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:38.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:29:38.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-02-05T06:29:39.574 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#1:9a09113a:::obj7:head# 2024-02-05T06:29:40.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:40.103 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-02-05T06:29:40.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T06:29:40.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-02-05T06:29:40.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:40.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:40.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:29:40.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:40.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:29:40.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-02-05T06:29:40.859 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#1:a557efb1:::obj8:head# 2024-02-05T06:29:41.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:41.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-02-05T06:29:41.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T06:29:41.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-02-05T06:29:41.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:41.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:41.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:29:41.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:41.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:29:41.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-02-05T06:29:42.186 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#1:fe351e27:::obj9:head# 2024-02-05T06:29:42.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:42.728 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-02-05T06:29:42.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T06:29:42.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-02-05T06:29:42.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:42.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:42.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:29:42.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:42.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:29:42.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-02-05T06:29:43.497 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#1:d1337354:::obj10:head# 2024-02-05T06:29:44.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:44.032 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-02-05T06:29:44.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T06:29:44.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-02-05T06:29:44.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:44.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:44.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:29:44.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:44.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:29:44.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-02-05T06:29:44.782 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-02-05T06:29:45.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:45.581 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-02-05T06:29:45.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T06:29:45.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-02-05T06:29:45.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:45.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:45.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:29:45.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:45.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:29:45.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-02-05T06:29:46.369 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-02-05T06:29:47.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:47.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-02-05T06:29:47.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T06:29:47.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-02-05T06:29:47.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:47.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:47.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:29:47.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:47.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:29:47.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-02-05T06:29:47.847 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-02-05T06:29:48.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:48.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-02-05T06:29:48.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T06:29:48.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-02-05T06:29:48.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:48.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:48.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:29:48.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:48.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:29:48.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-02-05T06:29:49.131 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#1:e0a44829:::obj14:head# 2024-02-05T06:29:49.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:49.667 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-02-05T06:29:49.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T06:29:49.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-02-05T06:29:49.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:49.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:49.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:29:49.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:49.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:29:49.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-02-05T06:29:50.343 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#1:ab946124:::obj15:head# 2024-02-05T06:29:50.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:50.880 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-02-05T06:29:50.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T06:29:50.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-02-05T06:29:50.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:50.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:50.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:29:50.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:50.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:29:50.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-02-05T06:29:51.629 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-02-05T06:29:52.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:52.169 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-02-05T06:29:52.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T06:29:52.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-02-05T06:29:52.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:52.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:52.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:29:52.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:52.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:29:52.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-02-05T06:29:52.905 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-02-05T06:29:53.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:53.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-02-05T06:29:53.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T06:29:53.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-02-05T06:29:53.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:53.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:53.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:29:53.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:53.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:29:53.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-02-05T06:29:54.149 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#1:b9836a99:::obj18:head# 2024-02-05T06:29:54.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:54.683 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-02-05T06:29:54.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T06:29:54.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-02-05T06:29:54.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:54.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:54.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:29:54.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:54.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:29:54.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-02-05T06:29:55.476 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-02-05T06:29:56.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:56.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-02-05T06:29:56.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T06:29:56.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-02-05T06:29:56.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:56.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:56.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:29:56.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:56.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:29:56.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-02-05T06:29:56.812 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#1:24b5611b:::obj20:head# 2024-02-05T06:29:57.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:57.348 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-02-05T06:29:57.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T06:29:57.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-02-05T06:29:57.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:57.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:57.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:29:57.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:57.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:29:57.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-02-05T06:29:58.056 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-02-05T06:29:58.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:58.591 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-02-05T06:29:58.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T06:29:58.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-02-05T06:29:58.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:58.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:58.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:29:58.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:58.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:29:58.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-02-05T06:29:59.417 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#1:c0c49def:::obj22:head# 2024-02-05T06:29:59.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:29:59.953 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-02-05T06:29:59.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T06:29:59.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-02-05T06:29:59.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:29:59.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:29:59.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:29:59.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:29:59.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:29:59.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-02-05T06:30:00.811 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#1:242c40e0:::obj23:head# 2024-02-05T06:30:01.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:30:01.361 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-02-05T06:30:01.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T06:30:01.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-02-05T06:30:01.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:30:01.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:30:01.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:30:01.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:30:01.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:30:01.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-02-05T06:30:02.080 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#1:7eacede5:::obj24:head# 2024-02-05T06:30:02.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:30:02.617 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-02-05T06:30:02.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T06:30:02.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-02-05T06:30:02.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:30:02.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:30:02.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T06:30:02.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:30:02.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:30:02.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-02-05T06:30:03.407 INFO:tasks.workunit.client.0.smithi148.stdout:remove 0#1:7109478f:::obj25:head# 2024-02-05T06:30:03.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T06:30:03.948 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-02-05T06:30:03.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T06:30:03.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-02-05T06:30:03.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:30:03.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:30:03.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:30:03.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:30:03.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:30:03.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-02-05T06:30:04.701 INFO:tasks.workunit.client.0.smithi148.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-02-05T06:30:05.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:30:05.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:30:05.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:30:05.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:30:05.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:30:05.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:30:05.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:30:05.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:30:05.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:30:05.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:30:05.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:30:05.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:30:05.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:30:05.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:30:05.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:30:05.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:30:05.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:30:05.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:30:05.246 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:30:05.246 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:30:05.246 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:30:05.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:30:05.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:30:05.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:30:05.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:30:05.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:30:05.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:30:05.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:30:05.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:30:05.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:30:05.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:30:05.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T06:30:05.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:30:05.249 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:30:05.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:30:05.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:30:05.250 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T06:30:05.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:30:05.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:30:05.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:30:05.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:30:05.284 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:05.284+0000 7f7dfb6be640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:30:05.287 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:05.288+0000 7f7dfb6be640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:30:05.289 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:05.288+0000 7f7dfb6be640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:30:05.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:30:05.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:30:05.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:30:05.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:30:05.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:30:05.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:05.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:30:05.570 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:30:05.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:05.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:30:05.823 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:05.824+0000 7f7dfb6be640 -1 Falling back to public interface 2024-02-05T06:30:05.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:06.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:06.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:06.896 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:30:06.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:30:06.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:06.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:30:07.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:08.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:08.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:08.215 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:30:08.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:30:08.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:08.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:30:08.297 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:08.295+0000 7f7dfb6be640 -1 osd.1 22 log_to_monitors true 2024-02-05T06:30:08.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:09.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:09.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:09.618 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:30:09.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:30:09.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:09.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:30:09.919 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [10,22) [v2:127.0.0.1:6802/2617047876,v1:127.0.0.1:6803/2617047876] [v2:127.0.0.1:6804/2617047876,v1:127.0.0.1:6805/2617047876] exists,up fde606da-da07-46a8-a1f9-b3fbf3d0e18d 2024-02-05T06:30:09.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:30:09.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:30:09.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:30:09.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:30:09.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:30:09.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:30:09.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:30:09.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:30:09.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:30:09.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:30:09.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:30:09.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:30:09.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:30:09.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:30:09.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:30:09.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:30:09.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:30:09.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:30:09.922 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:30:09.922 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:30:09.923 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:30:09.924 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:30:09.924 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:30:09.924 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:30:09.924 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:30:09.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:30:09.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:30:09.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:30:09.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:30:09.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:30:09.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:30:09.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:30:09.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:30:09.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:30:09.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T06:30:09.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:30:09.925 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:30:09.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:30:09.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T06:30:09.926 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:30:09.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:30:09.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:30:09.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:30:09.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:30:09.952 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:09.951+0000 7f0d948f6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:30:09.995 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:09.995+0000 7f0d948f6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:30:09.997 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:09.999+0000 7f0d948f6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:30:10.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:30:10.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:30:10.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:30:10.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:30:10.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:30:10.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:10.274 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:30:10.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:30:10.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:10.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:10.544 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:10.543+0000 7f0d948f6640 -1 Falling back to public interface 2024-02-05T06:30:10.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:11.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:11.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:11.615 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:30:11.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:30:11.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:11.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:11.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:12.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:12.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:12.920 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:30:12.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:30:12.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:12.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:13.178 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:13.179+0000 7f0d948f6640 -1 osd.0 21 log_to_monitors true 2024-02-05T06:30:13.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:14.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:14.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:14.345 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:30:14.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:30:14.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:14.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:14.815 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/2509897877,v1:127.0.0.1:6811/2509897877] [v2:127.0.0.1:6812/2509897877,v1:127.0.0.1:6813/2509897877] exists,up 63609b06-9d96-4822-b492-6b09cd860fc1 2024-02-05T06:30:14.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:30:14.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:30:14.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:30:14.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-02-05T06:30:14.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:30:14.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:30:14.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:30:14.817 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:30:14.817 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:30:14.817 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:30:14.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:30:14.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:30:14.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:30:14.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:30:14.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:30:14.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:30:14.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:30:14.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:30:14.945 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:30:15.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:30:15.256 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:30:15.256 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:30:15.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:30:15.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:30:15.257 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:30:15.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-02-05T06:30:15.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-02-05T06:30:15.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-02-05T06:30:15.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:30:15.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:30:15.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149699 2024-02-05T06:30:15.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149699 2024-02-05T06:30:15.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-111669149699' 2024-02-05T06:30:15.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:30:15.592 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:30:15.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-02-05T06:30:15.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-02-05T06:30:15.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-111669149699 2-64424509459' 2024-02-05T06:30:15.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:30:15.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-02-05T06:30:15.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:30:15.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:30:15.735 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-02-05T06:30:15.735 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:30:15.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-02-05T06:30:15.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-02-05T06:30:15.737 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 124554051586 2024-02-05T06:30:15.737 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:30:16.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-02-05T06:30:16.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:30:17.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:30:17.032 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:30:17.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-02-05T06:30:17.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:30:17.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149699 2024-02-05T06:30:17.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:30:17.366 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:30:17.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149699 2024-02-05T06:30:17.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:30:17.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149699 2024-02-05T06:30:17.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149699' 2024-02-05T06:30:17.369 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 111669149699 2024-02-05T06:30:17.370 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:30:17.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149699 -lt 111669149699 2024-02-05T06:30:17.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:30:17.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-02-05T06:30:17.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:30:17.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:30:17.696 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-02-05T06:30:17.696 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:30:17.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-02-05T06:30:17.697 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509459 2024-02-05T06:30:17.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-02-05T06:30:17.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:30:18.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-02-05T06:30:18.019 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:30:18.019 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:30:18.019 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:30:18.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:30:18.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:30:18.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:30:18.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:30:18.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:30:18.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:30:18.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:30:18.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:30:18.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:30:18.706 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:30:18.707 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:30:18.707 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:30:19.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:30:19.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:30:19.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:30:19.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-02-05T06:30:19.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-02-05T06:30:19.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-02-05T06:30:19.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:30:19.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:30:19.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:30:19.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:30:19.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T06:29:15.117860+0000 2024-02-05T06:30:19.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-02-05T06:30:19.696 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-02-05T06:30:19.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-02-05T06:29:15.117860+0000 2024-02-05T06:30:19.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:30:19.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:29:15.117860+0000 2024-02-05T06:30:19.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:30:19.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:30:19.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:30:19.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:30:19.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:30:19.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:30:19.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:30:19.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:30:20.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:29:15.117860+0000 '>' 2024-02-05T06:29:15.117860+0000 2024-02-05T06:30:20.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:30:21.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:30:21.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:30:21.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:30:21.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:30:21.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:30:21.022 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:30:21.022 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:30:21.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:29:15.117860+0000 '>' 2024-02-05T06:29:15.117860+0000 2024-02-05T06:30:21.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:30:22.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:30:22.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:30:22.372 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:30:22.372 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:30:22.372 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:30:22.373 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:30:22.373 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:30:22.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:29:15.117860+0000 '>' 2024-02-05T06:29:15.117860+0000 2024-02-05T06:30:22.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:30:23.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:30:23.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:30:23.693 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:30:23.693 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:30:23.693 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:30:23.694 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:30:23.694 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:30:24.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:29:15.117860+0000 '>' 2024-02-05T06:29:15.117860+0000 2024-02-05T06:30:24.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:30:25.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:30:25.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:30:25.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:30:25.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:30:25.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:30:25.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:30:25.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:30:25.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:30:20.436559+0000 '>' 2024-02-05T06:29:15.117860+0000 2024-02-05T06:30:25.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:30:25.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-02-05T06:30:25.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:30:25.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:30:25.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:30:25.325 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:30:25.326 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:30:25.326 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:30:25.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:30:25.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:30:25.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:30:25.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:30:25.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:30:25.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:30:25.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:30:25.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:30:25.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:30:25.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:30:25.744 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:30:25.744 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:30:25.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:30:25.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:30:25.745 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:30:25.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-02-05T06:30:25.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-02-05T06:30:25.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589' 2024-02-05T06:30:25.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:30:25.908 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:30:26.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-02-05T06:30:26.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-02-05T06:30:26.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-111669149702' 2024-02-05T06:30:26.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:30:26.048 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:30:26.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-02-05T06:30:26.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-02-05T06:30:26.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-111669149702 2-64424509462' 2024-02-05T06:30:26.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:30:26.216 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051589 2024-02-05T06:30:26.216 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:30:26.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:30:26.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051589 2024-02-05T06:30:26.218 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:30:26.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-02-05T06:30:26.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-02-05T06:30:26.220 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 124554051589 2024-02-05T06:30:26.220 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:30:26.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-02-05T06:30:26.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:30:27.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:30:27.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:30:27.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051589 2024-02-05T06:30:27.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:30:27.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149702 2024-02-05T06:30:27.819 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:30:27.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:30:27.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149702 2024-02-05T06:30:27.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:30:27.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-02-05T06:30:27.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149702' 2024-02-05T06:30:27.825 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 111669149702 2024-02-05T06:30:27.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:30:28.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-02-05T06:30:28.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:30:28.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-02-05T06:30:28.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:30:28.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:30:28.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-02-05T06:30:28.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:30:28.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-02-05T06:30:28.144 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509462 2024-02-05T06:30:28.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-02-05T06:30:28.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:30:28.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-02-05T06:30:28.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:30:28.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:30:28.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:30:28.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:30:28.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:30:28.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:30:28.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:30:28.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:30:28.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:30:28.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:30:28.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:30:29.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:30:29.249 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:30:29.250 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:30:29.250 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:30:29.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:30:29.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:30:29.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:30:29.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-02-05T06:30:29.926 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T06:30:29.926 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-02-05T06:30:21.814606+0000 21'30 30:128 [1,0,2] 1 [1,0,2] 1 21'30 2024-02-05T06:30:20.436559+0000 21'30 2024-02-05T06:30:20.436559+0000 0 1 periodic scrub scheduled @ 2024-02-06T06:30:20.436559+0000 17 0 2024-02-05T06:30:29.926 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T06:30:29.926 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T06:30:29.926 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:30:29.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-02-05T06:30:29.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:30:29.942 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:30:30.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:30:30.257 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:30:30.257 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:30:30.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:30:30.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:30:30.257 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:30:30.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051591 2024-02-05T06:30:30.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051591 2024-02-05T06:30:30.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591' 2024-02-05T06:30:30.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:30:30.424 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:30:30.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149704 2024-02-05T06:30:30.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149704 2024-02-05T06:30:30.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-111669149704' 2024-02-05T06:30:30.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:30:30.555 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:30:30.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-02-05T06:30:30.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-02-05T06:30:30.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-111669149704 2-64424509464' 2024-02-05T06:30:30.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:30:30.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051591 2024-02-05T06:30:30.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:30:30.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:30:30.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051591 2024-02-05T06:30:30.688 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:30:30.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051591 2024-02-05T06:30:30.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051591' 2024-02-05T06:30:30.689 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 124554051591 2024-02-05T06:30:30.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:30:30.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051591 -lt 124554051591 2024-02-05T06:30:30.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:30:30.988 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149704 2024-02-05T06:30:30.988 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:30:30.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:30:30.992 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149704 2024-02-05T06:30:30.992 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:30:30.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149704 2024-02-05T06:30:30.993 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 111669149704 2024-02-05T06:30:30.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149704' 2024-02-05T06:30:30.994 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:30:31.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149703 -lt 111669149704 2024-02-05T06:30:31.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:30:32.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:30:32.331 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:30:32.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149704 -lt 111669149704 2024-02-05T06:30:32.643 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:30:32.644 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-02-05T06:30:32.645 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:30:32.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:30:32.647 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-02-05T06:30:32.647 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:30:32.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-02-05T06:30:32.649 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509464 2024-02-05T06:30:32.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-02-05T06:30:32.649 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:30:32.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-02-05T06:30:32.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-02-05T06:30:32.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-02-05T06:30:33.110 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:30:33.111 INFO:tasks.workunit.client.0.smithi148.stdout: "num_bytes": 210, 2024-02-05T06:30:33.111 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects": 30, 2024-02-05T06:30:33.111 INFO:tasks.workunit.client.0.smithi148.stdout: "num_object_clones": 0, 2024-02-05T06:30:33.111 INFO:tasks.workunit.client.0.smithi148.stdout: "num_object_copies": 90, 2024-02-05T06:30:33.111 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_missing_on_primary": 0, 2024-02-05T06:30:33.111 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_missing": 0, 2024-02-05T06:30:33.111 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_degraded": 0, 2024-02-05T06:30:33.111 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_misplaced": 0, 2024-02-05T06:30:33.111 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_unfound": 0, 2024-02-05T06:30:33.111 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_dirty": 30, 2024-02-05T06:30:33.111 INFO:tasks.workunit.client.0.smithi148.stdout: "num_whiteouts": 0, 2024-02-05T06:30:33.111 INFO:tasks.workunit.client.0.smithi148.stdout: "num_read": 0, 2024-02-05T06:30:33.112 INFO:tasks.workunit.client.0.smithi148.stdout: "num_read_kb": 0, 2024-02-05T06:30:33.112 INFO:tasks.workunit.client.0.smithi148.stdout: "num_write": 30, 2024-02-05T06:30:33.112 INFO:tasks.workunit.client.0.smithi148.stdout: "num_write_kb": 30, 2024-02-05T06:30:33.112 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrub_errors": 0, 2024-02-05T06:30:33.112 INFO:tasks.workunit.client.0.smithi148.stdout: "num_shallow_scrub_errors": 0, 2024-02-05T06:30:33.112 INFO:tasks.workunit.client.0.smithi148.stdout: "num_deep_scrub_errors": 0, 2024-02-05T06:30:33.112 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_recovered": 26, 2024-02-05T06:30:33.112 INFO:tasks.workunit.client.0.smithi148.stdout: "num_bytes_recovered": 182, 2024-02-05T06:30:33.112 INFO:tasks.workunit.client.0.smithi148.stdout: "num_keys_recovered": 0, 2024-02-05T06:30:33.112 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_omap": 0, 2024-02-05T06:30:33.112 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_hit_set_archive": 0, 2024-02-05T06:30:33.112 INFO:tasks.workunit.client.0.smithi148.stdout: "num_bytes_hit_set_archive": 0, 2024-02-05T06:30:33.112 INFO:tasks.workunit.client.0.smithi148.stdout: "num_flush": 0, 2024-02-05T06:30:33.112 INFO:tasks.workunit.client.0.smithi148.stdout: "num_flush_kb": 0, 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout: "num_evict": 0, 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout: "num_evict_kb": 0, 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout: "num_promote": 0, 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout: "num_flush_mode_high": 0, 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout: "num_flush_mode_low": 0, 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout: "num_evict_mode_some": 0, 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout: "num_evict_mode_full": 0, 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_pinned": 0, 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout: "num_legacy_snapsets": 0, 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout: "num_large_omap_objects": 0, 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_manifest": 0, 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout: "num_omap_bytes": 0, 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout: "num_omap_keys": 0, 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout: "num_objects_repaired": 26 2024-02-05T06:30:33.113 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:30:33.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-02-05T06:30:33.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-02-05T06:30:33.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-02-05T06:30:33.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-02-05T06:30:33.283 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-02-05T06:30:33.284 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-02-05T06:30:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:30:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-02-05T06:30:33.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-02-05T06:30:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-02-05T06:30:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T06:30:33.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-02-05T06:30:33.560 INFO:tasks.workunit.client.0.smithi148.stderr:dumped all 2024-02-05T06:30:33.575 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:30:33.575 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:30:33.575 INFO:tasks.workunit.client.0.smithi148.stdout: "up_from": 29, 2024-02-05T06:30:33.575 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 124554051591, 2024-02-05T06:30:33.575 INFO:tasks.workunit.client.0.smithi148.stdout: "num_pgs": 1, 2024-02-05T06:30:33.575 INFO:tasks.workunit.client.0.smithi148.stdout: "num_osds": 1, 2024-02-05T06:30:33.576 INFO:tasks.workunit.client.0.smithi148.stdout: "num_per_pool_osds": 1, 2024-02-05T06:30:33.576 INFO:tasks.workunit.client.0.smithi148.stdout: "num_per_pool_omap_osds": 1, 2024-02-05T06:30:33.576 INFO:tasks.workunit.client.0.smithi148.stdout: "kb": 104857600, 2024-02-05T06:30:33.576 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used": 28472, 2024-02-05T06:30:33.576 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_data": 368, 2024-02-05T06:30:33.576 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_omap": 4, 2024-02-05T06:30:33.576 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_meta": 28091, 2024-02-05T06:30:33.576 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_avail": 104829128, 2024-02-05T06:30:33.576 INFO:tasks.workunit.client.0.smithi148.stdout: "statfs": { 2024-02-05T06:30:33.576 INFO:tasks.workunit.client.0.smithi148.stdout: "total": 107374182400, 2024-02-05T06:30:33.576 INFO:tasks.workunit.client.0.smithi148.stdout: "available": 107345027072, 2024-02-05T06:30:33.576 INFO:tasks.workunit.client.0.smithi148.stdout: "internally_reserved": 0, 2024-02-05T06:30:33.576 INFO:tasks.workunit.client.0.smithi148.stdout: "allocated": 376832, 2024-02-05T06:30:33.576 INFO:tasks.workunit.client.0.smithi148.stdout: "data_stored": 201819, 2024-02-05T06:30:33.577 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed": 0, 2024-02-05T06:30:33.577 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed_allocated": 0, 2024-02-05T06:30:33.577 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed_original": 0, 2024-02-05T06:30:33.577 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_allocated": 4402, 2024-02-05T06:30:33.577 INFO:tasks.workunit.client.0.smithi148.stdout: "internal_metadata": 28765902 2024-02-05T06:30:33.577 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:33.577 INFO:tasks.workunit.client.0.smithi148.stdout: "hb_peers": [ 2024-02-05T06:30:33.577 INFO:tasks.workunit.client.0.smithi148.stdout: 1, 2024-02-05T06:30:33.577 INFO:tasks.workunit.client.0.smithi148.stdout: 2 2024-02-05T06:30:33.577 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:30:33.577 INFO:tasks.workunit.client.0.smithi148.stdout: "snap_trim_queue_len": 0, 2024-02-05T06:30:33.577 INFO:tasks.workunit.client.0.smithi148.stdout: "num_snap_trimming": 0, 2024-02-05T06:30:33.577 INFO:tasks.workunit.client.0.smithi148.stdout: "num_shards_repaired": 13, 2024-02-05T06:30:33.577 INFO:tasks.workunit.client.0.smithi148.stdout: "op_queue_age_hist": { 2024-02-05T06:30:33.578 INFO:tasks.workunit.client.0.smithi148.stdout: "histogram": [], 2024-02-05T06:30:33.578 INFO:tasks.workunit.client.0.smithi148.stdout: "upper_bound": 1 2024-02-05T06:30:33.578 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:33.578 INFO:tasks.workunit.client.0.smithi148.stdout: "perf_stat": { 2024-02-05T06:30:33.578 INFO:tasks.workunit.client.0.smithi148.stdout: "commit_latency_ms": 0, 2024-02-05T06:30:33.578 INFO:tasks.workunit.client.0.smithi148.stdout: "apply_latency_ms": 0, 2024-02-05T06:30:33.578 INFO:tasks.workunit.client.0.smithi148.stdout: "commit_latency_ns": 0, 2024-02-05T06:30:33.578 INFO:tasks.workunit.client.0.smithi148.stdout: "apply_latency_ns": 0 2024-02-05T06:30:33.578 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:33.578 INFO:tasks.workunit.client.0.smithi148.stdout: "alerts": [], 2024-02-05T06:30:33.578 INFO:tasks.workunit.client.0.smithi148.stdout: "network_ping_times": [ 2024-02-05T06:30:33.578 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:30:33.578 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:30:33.578 INFO:tasks.workunit.client.0.smithi148.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T06:30:33.579 INFO:tasks.workunit.client.0.smithi148.stdout: "interfaces": [ 2024-02-05T06:30:33.579 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:30:33.579 INFO:tasks.workunit.client.0.smithi148.stdout: "interface": "back", 2024-02-05T06:30:33.579 INFO:tasks.workunit.client.0.smithi148.stdout: "average": { 2024-02-05T06:30:33.579 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:33.579 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:33.579 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:33.579 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:33.579 INFO:tasks.workunit.client.0.smithi148.stdout: "min": { 2024-02-05T06:30:33.579 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:33.579 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:33.579 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:33.579 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:33.579 INFO:tasks.workunit.client.0.smithi148.stdout: "max": { 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: "last": 0.187 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 2, 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: "interfaces": [ 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: "interface": "back", 2024-02-05T06:30:33.580 INFO:tasks.workunit.client.0.smithi148.stdout: "average": { 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: "min": { 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: "max": { 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:33.581 INFO:tasks.workunit.client.0.smithi148.stdout: "last": 0.299 2024-02-05T06:30:33.582 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:30:33.582 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:30:33.582 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:30:33.582 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:30:33.582 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:30:33.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T06:30:33.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-02-05T06:30:33.854 INFO:tasks.workunit.client.0.smithi148.stderr:dumped all 2024-02-05T06:30:33.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=13 2024-02-05T06:30:33.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 13 = 13 2024-02-05T06:30:33.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:30:33.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 1 = 0 -o 1 = 1 ']' 2024-02-05T06:30:33.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-02-05T06:30:33.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-02-05T06:30:33.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T06:30:33.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-02-05T06:30:34.135 INFO:tasks.workunit.client.0.smithi148.stderr:dumped all 2024-02-05T06:30:34.152 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:30:34.152 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:30:34.152 INFO:tasks.workunit.client.0.smithi148.stdout: "up_from": 26, 2024-02-05T06:30:34.152 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 111669149704, 2024-02-05T06:30:34.152 INFO:tasks.workunit.client.0.smithi148.stdout: "num_pgs": 1, 2024-02-05T06:30:34.153 INFO:tasks.workunit.client.0.smithi148.stdout: "num_osds": 1, 2024-02-05T06:30:34.153 INFO:tasks.workunit.client.0.smithi148.stdout: "num_per_pool_osds": 1, 2024-02-05T06:30:34.153 INFO:tasks.workunit.client.0.smithi148.stdout: "num_per_pool_omap_osds": 1, 2024-02-05T06:30:34.153 INFO:tasks.workunit.client.0.smithi148.stdout: "kb": 104857600, 2024-02-05T06:30:34.153 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used": 28472, 2024-02-05T06:30:34.153 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_data": 368, 2024-02-05T06:30:34.153 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_omap": 4, 2024-02-05T06:30:34.153 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_meta": 28091, 2024-02-05T06:30:34.153 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_avail": 104829128, 2024-02-05T06:30:34.153 INFO:tasks.workunit.client.0.smithi148.stdout: "statfs": { 2024-02-05T06:30:34.153 INFO:tasks.workunit.client.0.smithi148.stdout: "total": 107374182400, 2024-02-05T06:30:34.153 INFO:tasks.workunit.client.0.smithi148.stdout: "available": 107345027072, 2024-02-05T06:30:34.153 INFO:tasks.workunit.client.0.smithi148.stdout: "internally_reserved": 0, 2024-02-05T06:30:34.154 INFO:tasks.workunit.client.0.smithi148.stdout: "allocated": 376832, 2024-02-05T06:30:34.154 INFO:tasks.workunit.client.0.smithi148.stdout: "data_stored": 201819, 2024-02-05T06:30:34.154 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed": 0, 2024-02-05T06:30:34.154 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed_allocated": 0, 2024-02-05T06:30:34.154 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed_original": 0, 2024-02-05T06:30:34.154 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_allocated": 4398, 2024-02-05T06:30:34.154 INFO:tasks.workunit.client.0.smithi148.stdout: "internal_metadata": 28765906 2024-02-05T06:30:34.154 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.154 INFO:tasks.workunit.client.0.smithi148.stdout: "hb_peers": [ 2024-02-05T06:30:34.154 INFO:tasks.workunit.client.0.smithi148.stdout: 0, 2024-02-05T06:30:34.154 INFO:tasks.workunit.client.0.smithi148.stdout: 2 2024-02-05T06:30:34.154 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:30:34.154 INFO:tasks.workunit.client.0.smithi148.stdout: "snap_trim_queue_len": 0, 2024-02-05T06:30:34.154 INFO:tasks.workunit.client.0.smithi148.stdout: "num_snap_trimming": 0, 2024-02-05T06:30:34.155 INFO:tasks.workunit.client.0.smithi148.stdout: "num_shards_repaired": 13, 2024-02-05T06:30:34.155 INFO:tasks.workunit.client.0.smithi148.stdout: "op_queue_age_hist": { 2024-02-05T06:30:34.155 INFO:tasks.workunit.client.0.smithi148.stdout: "histogram": [], 2024-02-05T06:30:34.155 INFO:tasks.workunit.client.0.smithi148.stdout: "upper_bound": 1 2024-02-05T06:30:34.155 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.155 INFO:tasks.workunit.client.0.smithi148.stdout: "perf_stat": { 2024-02-05T06:30:34.155 INFO:tasks.workunit.client.0.smithi148.stdout: "commit_latency_ms": 0, 2024-02-05T06:30:34.155 INFO:tasks.workunit.client.0.smithi148.stdout: "apply_latency_ms": 0, 2024-02-05T06:30:34.155 INFO:tasks.workunit.client.0.smithi148.stdout: "commit_latency_ns": 0, 2024-02-05T06:30:34.155 INFO:tasks.workunit.client.0.smithi148.stdout: "apply_latency_ns": 0 2024-02-05T06:30:34.155 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.155 INFO:tasks.workunit.client.0.smithi148.stdout: "alerts": [], 2024-02-05T06:30:34.155 INFO:tasks.workunit.client.0.smithi148.stdout: "network_ping_times": [ 2024-02-05T06:30:34.156 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:30:34.156 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:30:34.156 INFO:tasks.workunit.client.0.smithi148.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T06:30:34.156 INFO:tasks.workunit.client.0.smithi148.stdout: "interfaces": [ 2024-02-05T06:30:34.156 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:30:34.156 INFO:tasks.workunit.client.0.smithi148.stdout: "interface": "back", 2024-02-05T06:30:34.156 INFO:tasks.workunit.client.0.smithi148.stdout: "average": { 2024-02-05T06:30:34.156 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:34.156 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:34.156 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:34.156 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.156 INFO:tasks.workunit.client.0.smithi148.stdout: "min": { 2024-02-05T06:30:34.156 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:34.157 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:34.157 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:34.157 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.157 INFO:tasks.workunit.client.0.smithi148.stdout: "max": { 2024-02-05T06:30:34.157 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:34.157 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:34.157 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:34.157 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.157 INFO:tasks.workunit.client.0.smithi148.stdout: "last": 0.27 2024-02-05T06:30:34.157 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:30:34.157 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:30:34.158 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.158 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:30:34.158 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 2, 2024-02-05T06:30:34.158 INFO:tasks.workunit.client.0.smithi148.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T06:30:34.158 INFO:tasks.workunit.client.0.smithi148.stdout: "interfaces": [ 2024-02-05T06:30:34.158 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:30:34.158 INFO:tasks.workunit.client.0.smithi148.stdout: "interface": "back", 2024-02-05T06:30:34.158 INFO:tasks.workunit.client.0.smithi148.stdout: "average": { 2024-02-05T06:30:34.158 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:34.158 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:34.158 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:34.158 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.158 INFO:tasks.workunit.client.0.smithi148.stdout: "min": { 2024-02-05T06:30:34.158 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:34.159 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:34.159 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:34.159 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.159 INFO:tasks.workunit.client.0.smithi148.stdout: "max": { 2024-02-05T06:30:34.159 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:34.159 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:34.159 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:34.159 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.159 INFO:tasks.workunit.client.0.smithi148.stdout: "last": 0.341 2024-02-05T06:30:34.159 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:30:34.159 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:30:34.159 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:30:34.159 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:30:34.159 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:30:34.160 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T06:30:34.160 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-02-05T06:30:34.448 INFO:tasks.workunit.client.0.smithi148.stderr:dumped all 2024-02-05T06:30:34.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=13 2024-02-05T06:30:34.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 13 = 13 2024-02-05T06:30:34.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:30:34.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 2 = 0 -o 2 = 1 ']' 2024-02-05T06:30:34.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: repair=0 2024-02-05T06:30:34.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T06:30:34.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 )' 2024-02-05T06:30:34.775 INFO:tasks.workunit.client.0.smithi148.stderr:dumped all 2024-02-05T06:30:34.790 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:30:34.790 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 2, 2024-02-05T06:30:34.790 INFO:tasks.workunit.client.0.smithi148.stdout: "up_from": 15, 2024-02-05T06:30:34.790 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 64424509465, 2024-02-05T06:30:34.790 INFO:tasks.workunit.client.0.smithi148.stdout: "num_pgs": 1, 2024-02-05T06:30:34.790 INFO:tasks.workunit.client.0.smithi148.stdout: "num_osds": 1, 2024-02-05T06:30:34.790 INFO:tasks.workunit.client.0.smithi148.stdout: "num_per_pool_osds": 1, 2024-02-05T06:30:34.790 INFO:tasks.workunit.client.0.smithi148.stdout: "num_per_pool_omap_osds": 1, 2024-02-05T06:30:34.790 INFO:tasks.workunit.client.0.smithi148.stdout: "kb": 104857600, 2024-02-05T06:30:34.790 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used": 27204, 2024-02-05T06:30:34.791 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_data": 368, 2024-02-05T06:30:34.791 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_omap": 1, 2024-02-05T06:30:34.791 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_meta": 26814, 2024-02-05T06:30:34.791 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_avail": 104830396, 2024-02-05T06:30:34.791 INFO:tasks.workunit.client.0.smithi148.stdout: "statfs": { 2024-02-05T06:30:34.791 INFO:tasks.workunit.client.0.smithi148.stdout: "total": 107374182400, 2024-02-05T06:30:34.791 INFO:tasks.workunit.client.0.smithi148.stdout: "available": 107346325504, 2024-02-05T06:30:34.791 INFO:tasks.workunit.client.0.smithi148.stdout: "internally_reserved": 0, 2024-02-05T06:30:34.791 INFO:tasks.workunit.client.0.smithi148.stdout: "allocated": 376832, 2024-02-05T06:30:34.791 INFO:tasks.workunit.client.0.smithi148.stdout: "data_stored": 201819, 2024-02-05T06:30:34.791 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed": 0, 2024-02-05T06:30:34.791 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed_allocated": 0, 2024-02-05T06:30:34.791 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed_original": 0, 2024-02-05T06:30:34.792 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_allocated": 1595, 2024-02-05T06:30:34.792 INFO:tasks.workunit.client.0.smithi148.stdout: "internal_metadata": 27457989 2024-02-05T06:30:34.792 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.792 INFO:tasks.workunit.client.0.smithi148.stdout: "hb_peers": [ 2024-02-05T06:30:34.792 INFO:tasks.workunit.client.0.smithi148.stdout: 0, 2024-02-05T06:30:34.792 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:30:34.792 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:30:34.792 INFO:tasks.workunit.client.0.smithi148.stdout: "snap_trim_queue_len": 0, 2024-02-05T06:30:34.792 INFO:tasks.workunit.client.0.smithi148.stdout: "num_snap_trimming": 0, 2024-02-05T06:30:34.792 INFO:tasks.workunit.client.0.smithi148.stdout: "num_shards_repaired": 0, 2024-02-05T06:30:34.792 INFO:tasks.workunit.client.0.smithi148.stdout: "op_queue_age_hist": { 2024-02-05T06:30:34.792 INFO:tasks.workunit.client.0.smithi148.stdout: "histogram": [], 2024-02-05T06:30:34.792 INFO:tasks.workunit.client.0.smithi148.stdout: "upper_bound": 1 2024-02-05T06:30:34.793 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.793 INFO:tasks.workunit.client.0.smithi148.stdout: "perf_stat": { 2024-02-05T06:30:34.793 INFO:tasks.workunit.client.0.smithi148.stdout: "commit_latency_ms": 0, 2024-02-05T06:30:34.793 INFO:tasks.workunit.client.0.smithi148.stdout: "apply_latency_ms": 0, 2024-02-05T06:30:34.793 INFO:tasks.workunit.client.0.smithi148.stdout: "commit_latency_ns": 0, 2024-02-05T06:30:34.793 INFO:tasks.workunit.client.0.smithi148.stdout: "apply_latency_ns": 0 2024-02-05T06:30:34.793 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.793 INFO:tasks.workunit.client.0.smithi148.stdout: "alerts": [], 2024-02-05T06:30:34.793 INFO:tasks.workunit.client.0.smithi148.stdout: "network_ping_times": [ 2024-02-05T06:30:34.793 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:30:34.793 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:30:34.793 INFO:tasks.workunit.client.0.smithi148.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T06:30:34.793 INFO:tasks.workunit.client.0.smithi148.stdout: "interfaces": [ 2024-02-05T06:30:34.793 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:30:34.794 INFO:tasks.workunit.client.0.smithi148.stdout: "interface": "back", 2024-02-05T06:30:34.794 INFO:tasks.workunit.client.0.smithi148.stdout: "average": { 2024-02-05T06:30:34.794 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:34.794 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:34.794 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:34.794 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.794 INFO:tasks.workunit.client.0.smithi148.stdout: "min": { 2024-02-05T06:30:34.794 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:34.794 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:34.794 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:34.794 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.794 INFO:tasks.workunit.client.0.smithi148.stdout: "max": { 2024-02-05T06:30:34.794 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:34.794 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:34.795 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:34.795 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.795 INFO:tasks.workunit.client.0.smithi148.stdout: "last": 0.39 2024-02-05T06:30:34.795 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:30:34.795 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:30:34.795 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.795 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:30:34.795 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:30:34.795 INFO:tasks.workunit.client.0.smithi148.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T06:30:34.795 INFO:tasks.workunit.client.0.smithi148.stdout: "interfaces": [ 2024-02-05T06:30:34.795 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:30:34.795 INFO:tasks.workunit.client.0.smithi148.stdout: "interface": "back", 2024-02-05T06:30:34.795 INFO:tasks.workunit.client.0.smithi148.stdout: "average": { 2024-02-05T06:30:34.796 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:34.796 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:34.796 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:34.796 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.796 INFO:tasks.workunit.client.0.smithi148.stdout: "min": { 2024-02-05T06:30:34.796 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:34.796 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:34.796 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:34.796 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.796 INFO:tasks.workunit.client.0.smithi148.stdout: "max": { 2024-02-05T06:30:34.796 INFO:tasks.workunit.client.0.smithi148.stdout: "1min": 0, 2024-02-05T06:30:34.796 INFO:tasks.workunit.client.0.smithi148.stdout: "5min": 0, 2024-02-05T06:30:34.796 INFO:tasks.workunit.client.0.smithi148.stdout: "15min": 0 2024-02-05T06:30:34.796 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:34.797 INFO:tasks.workunit.client.0.smithi148.stdout: "last": 0.349 2024-02-05T06:30:34.797 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:30:34.797 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:30:34.797 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:30:34.797 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:30:34.797 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:30:34.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 ).num_shards_repaired' 2024-02-05T06:30:34.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T06:30:35.080 INFO:tasks.workunit.client.0.smithi148.stderr:dumped all 2024-02-05T06:30:35.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=0 2024-02-05T06:30:35.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 0 = 0 2024-02-05T06:30:35.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T06:30:35.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum 2024-02-05T06:30:35.447 INFO:tasks.workunit.client.0.smithi148.stderr:dumped all 2024-02-05T06:30:35.464 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:30:35.464 INFO:tasks.workunit.client.0.smithi148.stdout: "up_from": 0, 2024-02-05T06:30:35.464 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 0, 2024-02-05T06:30:35.464 INFO:tasks.workunit.client.0.smithi148.stdout: "num_pgs": 3, 2024-02-05T06:30:35.464 INFO:tasks.workunit.client.0.smithi148.stdout: "num_osds": 3, 2024-02-05T06:30:35.465 INFO:tasks.workunit.client.0.smithi148.stdout: "num_per_pool_osds": 3, 2024-02-05T06:30:35.465 INFO:tasks.workunit.client.0.smithi148.stdout: "num_per_pool_omap_osds": 3, 2024-02-05T06:30:35.465 INFO:tasks.workunit.client.0.smithi148.stdout: "kb": 314572800, 2024-02-05T06:30:35.465 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used": 84148, 2024-02-05T06:30:35.465 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_data": 1104, 2024-02-05T06:30:35.465 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_omap": 10, 2024-02-05T06:30:35.465 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_used_meta": 82997, 2024-02-05T06:30:35.465 INFO:tasks.workunit.client.0.smithi148.stdout: "kb_avail": 314488652, 2024-02-05T06:30:35.465 INFO:tasks.workunit.client.0.smithi148.stdout: "statfs": { 2024-02-05T06:30:35.465 INFO:tasks.workunit.client.0.smithi148.stdout: "total": 322122547200, 2024-02-05T06:30:35.465 INFO:tasks.workunit.client.0.smithi148.stdout: "available": 322036379648, 2024-02-05T06:30:35.465 INFO:tasks.workunit.client.0.smithi148.stdout: "internally_reserved": 0, 2024-02-05T06:30:35.465 INFO:tasks.workunit.client.0.smithi148.stdout: "allocated": 1130496, 2024-02-05T06:30:35.465 INFO:tasks.workunit.client.0.smithi148.stdout: "data_stored": 605457, 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed": 0, 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed_allocated": 0, 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: "data_compressed_original": 0, 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_allocated": 10395, 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: "internal_metadata": 84989797 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: "hb_peers": [], 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: "snap_trim_queue_len": 0, 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: "num_snap_trimming": 0, 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: "num_shards_repaired": 26, 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: "op_queue_age_hist": { 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: "histogram": [], 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: "upper_bound": 1 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:35.466 INFO:tasks.workunit.client.0.smithi148.stdout: "perf_stat": { 2024-02-05T06:30:35.467 INFO:tasks.workunit.client.0.smithi148.stdout: "commit_latency_ms": 0, 2024-02-05T06:30:35.467 INFO:tasks.workunit.client.0.smithi148.stdout: "apply_latency_ms": 0, 2024-02-05T06:30:35.467 INFO:tasks.workunit.client.0.smithi148.stdout: "commit_latency_ns": 0, 2024-02-05T06:30:35.467 INFO:tasks.workunit.client.0.smithi148.stdout: "apply_latency_ns": 0 2024-02-05T06:30:35.467 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:30:35.467 INFO:tasks.workunit.client.0.smithi148.stdout: "alerts": [], 2024-02-05T06:30:35.467 INFO:tasks.workunit.client.0.smithi148.stdout: "network_ping_times": [] 2024-02-05T06:30:35.467 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:30:35.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T06:30:35.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-02-05T06:30:35.749 INFO:tasks.workunit.client.0.smithi148.stderr:dumped all 2024-02-05T06:30:35.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: COUNT=26 2024-02-05T06:30:35.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:794: TEST_repair_stats_ec: test 26 = 26 2024-02-05T06:30:35.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:30:35.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:30:35.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:30:35.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:30:35.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:30:35.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:30:35.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:30:35.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:30:35.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:30:36.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:30:36.081 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:30:36.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:30:36.083 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:30:36.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:30:36.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:30:36.085 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:30:36.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:30:36.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:30:36.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:30:36.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:30:36.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:30:36.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:30:36.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:30:36.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:30:36.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:30:36.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:30:36.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:30:36.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:30:36.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:30:36.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:30:36.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:30:36.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:30:36.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:30:36.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:30:36.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:30:36.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:30:36.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:30:36.118 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:30:36.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:30:36.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:30:36.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:30:36.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:30:36.121 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:30:36.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:30:36.123 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:30:36.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:30:36.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:30:36.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:30:36.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:30:36.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:30:36.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:30:36.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:30:36.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:30:36.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:30:36.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:30:36.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:30:36.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:30:36.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:30:36.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:30:36.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:30:36.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:30:36.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:30:36.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:30:36.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:30:36.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:30:36.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T06:30:36.137 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:30:36.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:30:36.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:30:36.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:30:36.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_request_scrub_priority td/osd-scrub-repair 2024-02-05T06:30:36.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6197: TEST_request_scrub_priority: local dir=td/osd-scrub-repair 2024-02-05T06:30:36.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6198: TEST_request_scrub_priority: local poolname=psr_pool 2024-02-05T06:30:36.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6199: TEST_request_scrub_priority: local objname=POBJ 2024-02-05T06:30:36.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6200: TEST_request_scrub_priority: local OBJECTS=64 2024-02-05T06:30:36.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6201: TEST_request_scrub_priority: local PGS=8 2024-02-05T06:30:36.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6203: TEST_request_scrub_priority: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T06:30:36.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:30:36.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:30:36.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:30:36.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:30:36.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:30:36.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T06:30:36.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:30:36.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:30:36.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:30:36.539 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:30:36.539 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:30:36.540 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:30:36.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:30:36.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T06:30:36.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:30:36.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:30:36.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:30:36.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:30:36.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:30:36.753 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:30:36.753 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:30:36.753 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:30:36.753 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:30:36.754 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:30:36.754 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:30:36.754 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:30:36.754 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:30:36.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:30:36.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T06:30:36.864 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:30:36.864 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:30:36.865 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:30:36.865 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:30:36.865 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:30:36.866 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:30:36.866 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:30:36.866 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:30:36.866 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:30:36.866 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:30:36.866 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:30:36.866 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:30:36.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:30:36.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T06:30:36.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6204: TEST_request_scrub_priority: run_mgr td/osd-scrub-repair x 2024-02-05T06:30:36.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:30:36.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:30:36.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:30:36.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:30:36.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:30:36.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:30:37.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:30:37.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:30:37.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:30:37.817 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:30:37.817 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:30:37.817 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:30:37.817 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:30:37.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:30:37.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:30:37.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6205: TEST_request_scrub_priority: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-02-05T06:30:37.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6206: TEST_request_scrub_priority: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-02-05T06:30:37.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6207: TEST_request_scrub_priority: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T06:30:37.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:30:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:30:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:30:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:30:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:30:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:30:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:30:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:30:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:30:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:30:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:30:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:30:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:30:37.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:30:37.859 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:30:37.859 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:30:37.859 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:30:37.859 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:30:37.859 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:30:37.859 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:30:37.859 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:30:37.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:30:37.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:30:37.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:30:37.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:30:37.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:30:37.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:30:37.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:30:37.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:30:37.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:30:37.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:30:37.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:30:37.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-02-05T06:30:37.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:30:37.861 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:30:37.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b539a1b-4d67-4ee9-b82a-1ebe70edc554 2024-02-05T06:30:37.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4b539a1b-4d67-4ee9-b82a-1ebe70edc554' 2024-02-05T06:30:37.861 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 4b539a1b-4d67-4ee9-b82a-1ebe70edc554 2024-02-05T06:30:37.861 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:30:37.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCNgMBl3U98NBAA8USVzB+EYhaIZjkDFWlnmg== 2024-02-05T06:30:37.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCNgMBl3U98NBAA8USVzB+EYhaIZjkDFWlnmg=="}' 2024-02-05T06:30:37.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b539a1b-4d67-4ee9-b82a-1ebe70edc554 -i td/osd-scrub-repair/0/new.json 2024-02-05T06:30:38.112 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:30:38.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:30:38.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQCNgMBl3U98NBAA8USVzB+EYhaIZjkDFWlnmg== --osd-uuid 4b539a1b-4d67-4ee9-b82a-1ebe70edc554 2024-02-05T06:30:38.166 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:38.167+0000 7fbf595cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:30:38.168 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:38.167+0000 7fbf595cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:30:38.170 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:38.171+0000 7fbf595cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:30:38.170 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:38.171+0000 7fbf595cb640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:30:38.170 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:38.171+0000 7fbf595cb640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:30:38.170 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:38.171+0000 7fbf595cb640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:30:38.246 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:38.247+0000 7fbf595cb640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:30:44.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:30:44.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:30:44.268 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:30:44.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:30:44.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:30:44.641 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:30:44.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:30:44.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:30:44.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:30:44.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:30:44.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T06:30:44.676 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:44.675+0000 7f7cb41d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:30:44.712 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:44.711+0000 7f7cb41d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:30:44.713 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:44.711+0000 7f7cb41d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:30:44.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:30:44.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:30:44.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:30:44.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:30:44.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:30:44.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:44.980 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:30:44.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:30:44.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:44.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:45.260 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:45.259+0000 7f7cb41d5640 -1 Falling back to public interface 2024-02-05T06:30:45.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:46.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:46.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:46.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:30:46.299 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:30:46.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:46.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:46.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:47.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:47.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:47.617 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:30:47.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:30:47.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:47.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:47.762 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:30:47.763+0000 7f7cb41d5640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:30:47.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:48.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:48.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:48.946 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:30:48.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:30:48.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:48.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:49.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:50.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:50.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:50.260 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:30:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:30:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:50.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:51.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:51.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:51.578 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:30:51.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:30:51.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:51.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:51.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:52.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:52.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:52.904 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:30:52.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:30:52.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:52.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:53.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:54.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:54.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:54.225 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:30:54.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:30:54.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:54.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:54.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:55.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:55.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:55.545 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:30:55.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:30:55.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:55.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:55.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:56.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:56.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:56.868 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:30:56.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:30:56.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:56.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:57.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:58.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:58.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:58.179 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:30:58.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:30:58.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:58.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:58.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:30:59.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:30:59.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:30:59.531 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:30:59.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:30:59.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:30:59.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:30:59.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:31:00.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:31:00.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:31:00.880 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:31:00.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:31:00.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:31:00.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:31:01.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:31:02.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:31:02.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:31:02.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:31:02.626 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:31:02.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:31:02.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:31:03.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:31:04.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:31:04.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:31:04.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:31:04.275 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:31:04.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:31:04.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:31:04.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:31:05.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:31:05.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:31:05.584 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:31:05.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:31:05.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:31:05.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:31:06.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:31:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:31:07.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:31:07.092 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:31:07.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:31:07.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:31:07.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:31:07.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:31:08.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:31:08.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:31:08.890 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:31:08.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:31:08.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:31:08.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:31:09.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:31:10.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:31:10.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:31:10.276 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:31:10.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:31:10.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:31:10.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:31:10.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:31:11.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:31:11.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:31:11.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:31:11.591 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:31:11.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:31:11.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:31:11.922 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2514584524,v1:127.0.0.1:6803/2514584524] [v2:127.0.0.1:6804/2514584524,v1:127.0.0.1:6805/2514584524] exists,up 4b539a1b-4d67-4ee9-b82a-1ebe70edc554 2024-02-05T06:31:11.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:31:11.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:31:11.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:31:11.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6209: TEST_request_scrub_priority: create_pool psr_pool 8 8 2024-02-05T06:31:11.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 8 8 2024-02-05T06:31:12.403 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'psr_pool' created 2024-02-05T06:31:12.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:31:13.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6210: TEST_request_scrub_priority: wait_for_clean 2024-02-05T06:31:13.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:31:13.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:31:13.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:31:13.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:31:13.431 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:31:13.432 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:31:13.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:31:13.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:31:13.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:31:13.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:31:13.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:31:13.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:31:13.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:31:13.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:31:13.571 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:31:13.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:31:13.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:31:13.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:31:13.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:31:14.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836485 2024-02-05T06:31:14.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836485 2024-02-05T06:31:14.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836485' 2024-02-05T06:31:14.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:31:14.065 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836485 2024-02-05T06:31:14.065 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:31:14.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:31:14.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836485 2024-02-05T06:31:14.067 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:31:14.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836485 2024-02-05T06:31:14.069 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836485 2024-02-05T06:31:14.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-02-05T06:31:14.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:31:14.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836485 2024-02-05T06:31:14.374 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:31:14.376 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:31:14.376 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:31:14.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-02-05T06:31:14.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:31:14.732 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:31:14.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:31:14.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:31:14.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:31:14.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:31:14.733 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:31:15.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-02-05T06:31:15.063 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:31:15.063 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:31:15.063 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:31:15.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-02-05T06:31:15.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:31:15.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:31:15.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6212: TEST_request_scrub_priority: local osd=0 2024-02-05T06:31:15.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6213: TEST_request_scrub_priority: add_something td/osd-scrub-repair psr_pool POBJ noscrub 2024-02-05T06:31:15.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:31:15.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-02-05T06:31:15.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-02-05T06:31:15.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:31:15.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:31:15.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:31:15.839 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:31:15.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:31:16.391 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:31:16.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:31:16.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:31:16.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-02-05T06:31:16.630 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6214: TEST_request_scrub_priority: get_primary psr_pool POBJ 2024-02-05T06:31:16.631 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-02-05T06:31:16.631 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-02-05T06:31:16.631 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-02-05T06:31:16.631 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:31:16.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6214: TEST_request_scrub_priority: local primary=0 2024-02-05T06:31:16.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6215: TEST_request_scrub_priority: get_pg psr_pool POBJ 2024-02-05T06:31:16.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-02-05T06:31:16.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-02-05T06:31:16.952 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-02-05T06:31:16.952 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:31:17.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6215: TEST_request_scrub_priority: local pg=1.1 2024-02-05T06:31:17.252 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: ceph osd dump 2024-02-05T06:31:17.253 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: grep '^pool.*['\'']psr_pool['\'']' 2024-02-05T06:31:17.253 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: awk '{ print $2 }' 2024-02-05T06:31:17.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: poolid=1 2024-02-05T06:31:17.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6218: TEST_request_scrub_priority: local otherpgs 2024-02-05T06:31:17.579 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: expr 8 - 1 2024-02-05T06:31:17.580 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: seq 0 7 2024-02-05T06:31:17.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T06:31:17.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.0 2024-02-05T06:31:17.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.0 = 1.1 ']' 2024-02-05T06:31:17.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 ' 2024-02-05T06:31:17.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T06:31:17.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T06:31:17.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:17.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T06:31:17.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:17.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:17.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.0 schedule-scrub 1.0 2024-02-05T06:31:18.043 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:31:18.044 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:31:18.044 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:31:18.044 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-29T06:29:38.047081+0000" 2024-02-05T06:31:18.044 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:31:18.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T06:31:18.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.1 2024-02-05T06:31:18.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.1 = 1.1 ']' 2024-02-05T06:31:18.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6223: TEST_request_scrub_priority: continue 2024-02-05T06:31:18.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T06:31:18.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.2 2024-02-05T06:31:18.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.2 = 1.1 ']' 2024-02-05T06:31:18.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 ' 2024-02-05T06:31:18.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T06:31:18.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T06:31:18.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:18.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:18.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T06:31:18.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:18.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.2 schedule-scrub 1.2 2024-02-05T06:31:18.498 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:31:18.498 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:31:18.498 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:31:18.499 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-29T06:29:38.501987+0000" 2024-02-05T06:31:18.499 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:31:18.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T06:31:18.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.3 2024-02-05T06:31:18.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.3 = 1.1 ']' 2024-02-05T06:31:18.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 ' 2024-02-05T06:31:18.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T06:31:18.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T06:31:18.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:18.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:18.515 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T06:31:18.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:18.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.3 schedule-scrub 1.3 2024-02-05T06:31:18.969 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:31:18.969 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:31:18.969 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:31:18.969 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-29T06:29:38.972322+0000" 2024-02-05T06:31:18.969 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:31:18.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T06:31:18.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.4 2024-02-05T06:31:18.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.4 = 1.1 ']' 2024-02-05T06:31:18.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 ' 2024-02-05T06:31:18.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T06:31:18.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T06:31:18.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:18.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T06:31:18.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:19.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:19.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.4 schedule-scrub 1.4 2024-02-05T06:31:19.424 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:31:19.425 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:31:19.425 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:31:19.425 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-29T06:29:39.428254+0000" 2024-02-05T06:31:19.425 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:31:19.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T06:31:19.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.5 2024-02-05T06:31:19.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.5 = 1.1 ']' 2024-02-05T06:31:19.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 ' 2024-02-05T06:31:19.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T06:31:19.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T06:31:19.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:19.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:19.440 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T06:31:19.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:19.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.5 schedule-scrub 1.5 2024-02-05T06:31:19.843 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:31:19.843 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:31:19.843 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:31:19.843 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-29T06:29:39.846696+0000" 2024-02-05T06:31:19.843 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:31:19.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T06:31:19.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.6 2024-02-05T06:31:19.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.6 = 1.1 ']' 2024-02-05T06:31:19.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 ' 2024-02-05T06:31:19.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T06:31:19.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T06:31:19.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:19.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:19.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T06:31:20.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:20.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.6 schedule-scrub 1.6 2024-02-05T06:31:20.280 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:31:20.280 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:31:20.280 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:31:20.280 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-29T06:29:40.283848+0000" 2024-02-05T06:31:20.280 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:31:20.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T06:31:20.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.7 2024-02-05T06:31:20.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.7 = 1.1 ']' 2024-02-05T06:31:20.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 1.7 ' 2024-02-05T06:31:20.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T06:31:20.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T06:31:20.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:20.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:20.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T06:31:20.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:20.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.7 schedule-scrub 1.7 2024-02-05T06:31:20.703 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:31:20.703 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:31:20.704 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:31:20.704 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-29T06:29:40.707203+0000" 2024-02-05T06:31:20.704 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:31:20.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: sleep 15 2024-02-05T06:31:35.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6232: TEST_request_scrub_priority: flush_pg_stats 2024-02-05T06:31:35.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:31:35.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:31:36.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:31:36.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:31:36.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:31:36.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:31:36.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836490 2024-02-05T06:31:36.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836490 2024-02-05T06:31:36.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490' 2024-02-05T06:31:36.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:31:36.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836490 2024-02-05T06:31:36.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:31:36.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:31:36.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836490 2024-02-05T06:31:36.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:31:36.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836490 2024-02-05T06:31:36.194 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836490 2024-02-05T06:31:36.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-02-05T06:31:36.194 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:31:36.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836490 2024-02-05T06:31:36.503 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6235: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T06:31:36.503 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T06:31:36.503 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:36.503 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:36.504 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T06:31:36.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6235: TEST_request_scrub_priority: local last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:36.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6236: TEST_request_scrub_priority: ceph tell 1.1 scrub 2024-02-05T06:31:36.974 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:31:36.974 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:31:36.974 INFO:tasks.workunit.client.0.smithi148.stdout: "must": true, 2024-02-05T06:31:36.974 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "0.000000" 2024-02-05T06:31:36.974 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:31:36.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: ceph osd unset noscrub 2024-02-05T06:31:37.389 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is unset 2024-02-05T06:31:37.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6239: TEST_request_scrub_priority: ceph osd unset nodeep-scrub 2024-02-05T06:31:37.959 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is unset 2024-02-05T06:31:37.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6241: TEST_request_scrub_priority: wait_for_scrub 1.1 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:37.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.1 2024-02-05T06:31:37.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:37.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:31:37.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:31:37.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:37.984 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-02-05T06:31:37.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T06:31:37.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:37.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:37.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T06:31:38.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:31:12.331266+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:38.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:31:39.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:31:39.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:39.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-02-05T06:31:39.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T06:31:39.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:39.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:39.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T06:31:39.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:31:12.331266+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:39.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:31:40.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:31:40.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:40.653 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-02-05T06:31:40.654 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T06:31:40.654 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:40.654 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:40.655 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T06:31:40.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:31:12.331266+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:40.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:31:41.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:31:41.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:41.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-02-05T06:31:41.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T06:31:41.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:41.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:41.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T06:31:42.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:31:37.585635+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:42.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:31:42.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T06:31:42.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.0 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:42.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:31:42.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:42.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:31:42.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:31:42.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:42.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:31:42.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:31:42.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:42.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:42.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:31:42.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:31:38.576421+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:42.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:31:42.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T06:31:42.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.2 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:42.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.2 2024-02-05T06:31:42.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:42.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:31:42.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:31:42.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:42.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-02-05T06:31:42.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-02-05T06:31:42.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:42.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:42.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-02-05T06:31:42.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:31:39.527826+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:42.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:31:42.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T06:31:42.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.3 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:42.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-02-05T06:31:42.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:42.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:31:42.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:31:42.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:42.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T06:31:42.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T06:31:42.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:42.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:42.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T06:31:43.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:31:40.510898+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:43.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:31:43.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T06:31:43.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.4 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:43.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.4 2024-02-05T06:31:43.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:43.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:31:43.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:31:43.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:43.176 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-02-05T06:31:43.176 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-02-05T06:31:43.176 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:43.176 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:43.176 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-02-05T06:31:43.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-29T06:29:39.428254+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:43.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:31:44.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:31:44.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:44.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-02-05T06:31:44.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-02-05T06:31:44.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:44.475 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:44.476 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-02-05T06:31:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-29T06:29:39.428254+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:44.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:31:45.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:31:45.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:45.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-02-05T06:31:45.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-02-05T06:31:45.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:45.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:45.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-02-05T06:31:46.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-29T06:29:39.428254+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:46.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:31:47.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:31:47.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:47.095 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-02-05T06:31:47.095 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-02-05T06:31:47.095 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:47.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:47.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-02-05T06:31:47.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:31:41.473171+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:47.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:31:47.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T06:31:47.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.5 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:47.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.5 2024-02-05T06:31:47.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:47.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:31:47.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:31:47.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:47.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-02-05T06:31:47.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-02-05T06:31:47.412 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:47.413 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:47.413 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-02-05T06:31:47.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:31:42.468837+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:47.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:31:47.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T06:31:47.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.6 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:47.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.6 2024-02-05T06:31:47.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:47.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:31:47.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:31:47.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:47.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-02-05T06:31:47.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-02-05T06:31:47.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:47.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-02-05T06:31:47.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:48.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:31:43.512489+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:48.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:31:48.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T06:31:48.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.7 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:48.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.7 2024-02-05T06:31:48.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:48.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:31:48.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:31:48.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:48.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-02-05T06:31:48.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-02-05T06:31:48.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:48.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:48.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-02-05T06:31:48.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:31:44.561380+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:48.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:31:48.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T06:31:48.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.1 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:48.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.1 2024-02-05T06:31:48.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:48.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:31:48.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:31:48.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:31:48.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-02-05T06:31:48.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T06:31:48.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:31:48.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:31:48.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T06:31:48.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:31:37.585635+0000 '>' 2024-02-05T06:31:12.331266+0000 2024-02-05T06:31:48.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:31:48.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: grep -v purged_snaps 2024-02-05T06:31:48.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: grep 'log_channel.*scrub ok' td/osd-scrub-repair/osd.0.log 2024-02-05T06:31:48.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: head -1 2024-02-05T06:31:48.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: sed 's/.*[[]DBG[]]//' 2024-02-05T06:31:48.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: grep -q 1.1 2024-02-05T06:31:48.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:31:48.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:31:48.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:31:48.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:31:48.627 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:31:48.627 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:31:48.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:31:48.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:31:48.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:31:48.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:31:48.938 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:31:48.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:31:48.941 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:31:48.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:31:48.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:31:48.943 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:31:48.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:31:48.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:31:48.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:31:48.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:31:48.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:31:48.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:31:48.949 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:31:48.960 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:31:48.960 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:31:48.960 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:31:48.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:31:48.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:31:48.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:31:48.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:31:48.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:31:48.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:31:48.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:31:48.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:31:48.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:31:48.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:31:48.964 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:31:48.964 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:31:48.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:31:48.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:31:48.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:31:48.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:31:48.967 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:31:48.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:31:48.968 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:31:48.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:31:48.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:31:48.970 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:31:48.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:31:48.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:31:48.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:31:48.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:31:48.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:31:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:31:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:31:48.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:31:48.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:31:48.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:31:48.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:31:48.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:31:48.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:31:48.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:31:48.980 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:31:48.980 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:31:48.980 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:31:48.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T06:31:48.981 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:31:48.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:31:48.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:31:48.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:31:48.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_scrub_warning td/osd-scrub-repair 2024-02-05T06:31:48.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5842: TEST_scrub_warning: local dir=td/osd-scrub-repair 2024-02-05T06:31:48.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5843: TEST_scrub_warning: local poolname=psr_pool 2024-02-05T06:31:48.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5844: TEST_scrub_warning: local objname=POBJ 2024-02-05T06:31:48.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5845: TEST_scrub_warning: local scrubs=5 2024-02-05T06:31:48.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5846: TEST_scrub_warning: local deep_scrubs=5 2024-02-05T06:31:48.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5847: TEST_scrub_warning: local i1_day=86400 2024-02-05T06:31:48.984 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5848: TEST_scrub_warning: calc 86400 '*' 7 2024-02-05T06:31:48.984 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 86400 * 7}' 2024-02-05T06:31:48.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5848: TEST_scrub_warning: local i7_days=604800 2024-02-05T06:31:48.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5849: TEST_scrub_warning: calc 86400 '*' 14 2024-02-05T06:31:48.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 86400 * 14}' 2024-02-05T06:31:48.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5849: TEST_scrub_warning: local i14_days=1209600 2024-02-05T06:31:48.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5850: TEST_scrub_warning: local overdue=0.5 2024-02-05T06:31:48.988 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_scrub_warning: calc 604800 + 86400 + '(' 604800 '*' 0.5 ')' 2024-02-05T06:31:48.988 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 604800 + 86400 + ( 604800 * 0.5 )}' 2024-02-05T06:31:48.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_scrub_warning: local conf_overdue_seconds=993600 2024-02-05T06:31:48.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5852: TEST_scrub_warning: calc 1209600 + 86400 + '(' 1209600 '*' 0.5 ')' 2024-02-05T06:31:48.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 1209600 + 86400 + ( 1209600 * 0.5 )}' 2024-02-05T06:31:48.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5852: TEST_scrub_warning: local pool_overdue_seconds=1900800 2024-02-05T06:31:48.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5854: TEST_scrub_warning: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T06:31:48.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:31:48.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:31:48.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:31:48.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:31:48.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:31:48.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T06:31:49.393 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:31:49.394 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:31:49.394 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:31:49.394 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:31:49.394 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:31:49.394 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:31:49.395 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:31:49.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T06:31:49.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:31:49.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:31:49.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:31:49.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:31:49.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:31:49.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:31:49.600 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:31:49.600 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:31:49.600 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:31:49.600 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:31:49.601 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:31:49.601 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:31:49.601 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:31:49.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:31:49.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T06:31:49.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:31:49.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:31:49.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:31:49.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:31:49.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:31:49.720 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:31:49.720 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:31:49.720 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:31:49.720 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:31:49.720 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:31:49.720 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:31:49.721 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:31:49.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:31:49.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T06:31:49.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5855: TEST_scrub_warning: run_mgr td/osd-scrub-repair x --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2024-02-05T06:31:49.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:31:49.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:31:49.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:31:49.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:31:49.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:31:49.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:31:50.705 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:31:50.705 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:31:50.705 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:31:50.705 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:31:50.705 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:31:50.706 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:31:50.706 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:31:50.707 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:31:50.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2024-02-05T06:31:50.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5856: TEST_scrub_warning: run_osd td/osd-scrub-repair 0 --osd_scrub_backoff_ratio=0 2024-02-05T06:31:50.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:31:50.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:31:50.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:31:50.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:31:50.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:31:50.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:31:50.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:31:50.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:31:50.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:31:50.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:31:50.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:31:50.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:31:50.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:31:50.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:31:50.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:31:50.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:31:50.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:31:50.746 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:31:50.746 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:31:50.746 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:31:50.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:31:50.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:31:50.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:31:50.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:31:50.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:31:50.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:31:50.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:31:50.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:31:50.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:31:50.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:31:50.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:31:50.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:31:50.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_backoff_ratio=0 2024-02-05T06:31:50.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:31:50.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:31:50.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f289ab33-7898-4f4f-a625-1549365c06a8 2024-02-05T06:31:50.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f289ab33-7898-4f4f-a625-1549365c06a8' 2024-02-05T06:31:50.749 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 f289ab33-7898-4f4f-a625-1549365c06a8 2024-02-05T06:31:50.750 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:31:50.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWgMBlI6rQLRAAQpk9a8n7B+aVchOlZ/c88w== 2024-02-05T06:31:50.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWgMBlI6rQLRAAQpk9a8n7B+aVchOlZ/c88w=="}' 2024-02-05T06:31:50.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f289ab33-7898-4f4f-a625-1549365c06a8 -i td/osd-scrub-repair/0/new.json 2024-02-05T06:31:51.050 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:31:51.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:31:51.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 --mkfs --key AQDWgMBlI6rQLRAAQpk9a8n7B+aVchOlZ/c88w== --osd-uuid f289ab33-7898-4f4f-a625-1549365c06a8 2024-02-05T06:31:51.100 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:31:51.101+0000 7efcb9429640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:31:51.143 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:31:51.141+0000 7efcb9429640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:31:51.144 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:31:51.145+0000 7efcb9429640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:31:51.145 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:31:51.145+0000 7efcb9429640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:31:51.145 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:31:51.145+0000 7efcb9429640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:31:51.145 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:31:51.145+0000 7efcb9429640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:31:51.234 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:31:51.233+0000 7efcb9429640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:31:57.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:31:57.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:31:57.306 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:31:57.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:31:57.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:31:57.686 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:31:57.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:31:57.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 2024-02-05T06:31:57.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:31:57.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:31:57.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:31:57.719 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:31:57.717+0000 7f97b4ebd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:31:57.750 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:31:57.749+0000 7f97b4ebd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:31:57.752 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:31:57.753+0000 7f97b4ebd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:31:58.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:31:58.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:31:58.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:31:58.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:31:58.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:31:58.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:31:58.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:31:58.017 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:31:58.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:31:58.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:31:58.293 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:31:58.293+0000 7f97b4ebd640 -1 Falling back to public interface 2024-02-05T06:31:58.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:31:59.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:31:59.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:31:59.359 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:31:59.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:31:59.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:31:59.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:31:59.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:00.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:00.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:00.671 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:32:00.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:32:00.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:00.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:00.810 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:32:00.809+0000 7f97b4ebd640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:32:00.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:01.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:01.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:01.992 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:32:01.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:32:01.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:01.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:02.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:03.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:03.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:03.316 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:32:03.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:32:03.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:03.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:03.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:04.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:04.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:04.777 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:32:04.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:32:04.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:04.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:05.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:06.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:06.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:06.093 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:32:06.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:32:06.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:06.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:06.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:07.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:07.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:07.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:32:07.400 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:32:07.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:07.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:07.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:08.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:08.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:08.767 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:32:08.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:32:08.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:08.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:09.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:10.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:10.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:10.081 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:32:10.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:32:10.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:10.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:10.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:11.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:11.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:11.391 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:32:11.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:32:11.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:11.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:11.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:12.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:12.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:12.781 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:32:12.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:32:12.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:12.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:13.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:14.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:14.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:14.087 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:32:14.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:32:14.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:14.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:14.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:15.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:15.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:15.709 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:32:15.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:32:15.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:15.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:16.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:17.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:17.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:17.401 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:32:17.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:32:17.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:17.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:18.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:19.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:19.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:19.308 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:32:19.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:32:19.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:19.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:20.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:21.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:21.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:21.062 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:32:21.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:32:21.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:21.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:21.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:22.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:22.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:22.521 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:32:22.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:32:22.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:22.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:23.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:32:24.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:32:24.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:32:24.174 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:32:24.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:32:24.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:32:24.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:32:24.573 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/781497063,v1:127.0.0.1:6803/781497063] [v2:127.0.0.1:6804/781497063,v1:127.0.0.1:6805/781497063] exists,up f289ab33-7898-4f4f-a625-1549365c06a8 2024-02-05T06:32:24.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:32:24.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:32:24.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:32:24.575 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: expr 5 + 5 2024-02-05T06:32:24.576 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: seq 1 10 2024-02-05T06:32:24.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:32:24.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-1 1 1 2024-02-05T06:32:24.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-1 1 1 2024-02-05T06:32:25.116 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'psr_pool-1' created 2024-02-05T06:32:25.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:32:26.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:32:26.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:32:26.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:32:26.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:32:26.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:32:26.142 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:32:26.142 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:32:26.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:32:26.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:32:26.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:32:26.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:32:26.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:32:26.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:32:26.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:32:26.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:32:26.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:32:26.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:32:26.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:32:26.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:32:26.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:32:26.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-02-05T06:32:26.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-02-05T06:32:26.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-02-05T06:32:26.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:32:26.773 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-02-05T06:32:26.774 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:32:26.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:32:26.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-02-05T06:32:26.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:32:26.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-02-05T06:32:26.778 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836487 2024-02-05T06:32:26.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-02-05T06:32:26.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:27.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 0 -lt 21474836487 2024-02-05T06:32:27.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:32:28.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:32:28.085 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:28.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836487 2024-02-05T06:32:28.399 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:32:28.400 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:32:28.400 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:32:28.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:32:28.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:32:28.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:32:28.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:32:28.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:32:28.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:32:28.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:32:28.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:32:29.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:32:29.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:32:29.097 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:32:29.097 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:32:29.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:32:29.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:32:29.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:32:29.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 1 = 1 ']' 2024-02-05T06:32:29.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: ceph osd pool set psr_pool-1 scrub_max_interval 1209600 2024-02-05T06:32:29.944 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 1 scrub_max_interval to 1209600 2024-02-05T06:32:29.962 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:32:29.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 1 = 6 ']' 2024-02-05T06:32:29.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:32:29.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-2 1 1 2024-02-05T06:32:29.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-2 1 1 2024-02-05T06:32:30.370 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'psr_pool-2' created 2024-02-05T06:32:30.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:32:31.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:32:31.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:32:31.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:32:31.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:32:31.392 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:32:31.393 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:32:31.393 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:32:31.393 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:32:31.393 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:32:31.394 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:32:31.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:32:31.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:32:31.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:32:31.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:32:31.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:32:31.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:32:31.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:32:31.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:32:31.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:32:31.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:32:32.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-02-05T06:32:32.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-02-05T06:32:32.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-02-05T06:32:32.010 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:32:32.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-02-05T06:32:32.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:32:32.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:32:32.012 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-02-05T06:32:32.013 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:32:32.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-02-05T06:32:32.014 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836489 2024-02-05T06:32:32.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-02-05T06:32:32.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:32.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2024-02-05T06:32:32.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:32:33.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:32:33.296 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:33.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-02-05T06:32:33.597 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:32:33.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:32:33.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:32:33.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-02-05T06:32:34.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:32:34.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:32:34.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:32:34.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:32:34.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:32:34.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:32:34.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:32:34.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T06:32:34.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:32:34.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:32:34.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:32:34.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-02-05T06:32:34.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:32:34.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:32:34.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 2 = 1 ']' 2024-02-05T06:32:34.710 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:32:34.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 2 = 6 ']' 2024-02-05T06:32:34.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:32:34.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-3 1 1 2024-02-05T06:32:34.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-3 1 1 2024-02-05T06:32:35.248 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'psr_pool-3' created 2024-02-05T06:32:35.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:32:36.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:32:36.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:32:36.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:32:36.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:32:36.279 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:32:36.279 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:32:36.279 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:32:36.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:32:36.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:32:36.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:32:36.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:32:36.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:32:36.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:32:36.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:32:36.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:32:36.421 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:32:36.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:32:36.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:32:36.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:32:36.735 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:32:36.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-02-05T06:32:36.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-02-05T06:32:36.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-02-05T06:32:36.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:32:36.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:32:36.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-02-05T06:32:36.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:32:36.892 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-02-05T06:32:36.892 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:32:36.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-02-05T06:32:36.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-02-05T06:32:36.894 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836491 2024-02-05T06:32:36.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:37.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836491 2024-02-05T06:32:37.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:32:38.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:32:38.195 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:38.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-02-05T06:32:38.513 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:32:38.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:32:38.514 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:32:38.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-02-05T06:32:38.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:32:38.906 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:32:38.906 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:32:38.906 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:32:38.906 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:32:38.906 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:32:38.906 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:32:39.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:32:39.229 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:32:39.229 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:32:39.230 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:32:39.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-02-05T06:32:39.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:32:39.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:32:39.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 3 = 1 ']' 2024-02-05T06:32:39.587 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:32:39.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 3 = 6 ']' 2024-02-05T06:32:39.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:32:39.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-4 1 1 2024-02-05T06:32:39.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-4 1 1 2024-02-05T06:32:39.985 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'psr_pool-4' created 2024-02-05T06:32:40.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:32:41.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:32:41.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:32:41.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:32:41.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:32:41.008 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:32:41.008 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:32:41.009 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:32:41.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:32:41.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:32:41.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:32:41.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:32:41.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:32:41.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:32:41.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:32:41.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:32:41.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:32:41.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:32:41.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:32:41.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:32:41.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:32:41.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-02-05T06:32:41.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-02-05T06:32:41.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-02-05T06:32:41.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:32:41.629 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-02-05T06:32:41.629 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:32:41.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:32:41.631 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-02-05T06:32:41.631 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:32:41.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-02-05T06:32:41.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-02-05T06:32:41.632 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836493 2024-02-05T06:32:41.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:41.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-02-05T06:32:41.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:32:42.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:32:42.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:43.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-02-05T06:32:43.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:32:44.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:32:44.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:44.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-02-05T06:32:44.564 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:32:44.565 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:32:44.565 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:32:44.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T06:32:44.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:32:44.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:32:44.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:32:44.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:32:44.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:32:44.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:32:44.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:32:45.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:32:45.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:32:45.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:32:45.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:32:45.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T06:32:45.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:32:45.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:32:45.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 4 = 1 ']' 2024-02-05T06:32:45.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:32:45.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 4 = 6 ']' 2024-02-05T06:32:45.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:32:45.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-5 1 1 2024-02-05T06:32:45.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-5 1 1 2024-02-05T06:32:46.180 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'psr_pool-5' created 2024-02-05T06:32:46.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:32:47.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:32:47.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:32:47.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:32:47.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:32:47.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:32:47.207 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:32:47.207 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:32:47.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:32:47.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:32:47.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:32:47.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:32:47.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:32:47.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:32:47.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:32:47.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:32:47.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:32:47.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:32:47.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:32:47.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:32:47.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:32:47.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-02-05T06:32:47.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-02-05T06:32:47.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-02-05T06:32:47.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:32:47.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-02-05T06:32:47.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:32:47.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:32:47.833 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-02-05T06:32:47.834 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:32:47.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-02-05T06:32:47.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-02-05T06:32:47.835 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836495 2024-02-05T06:32:47.836 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:48.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-02-05T06:32:48.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:32:49.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:32:49.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:49.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-02-05T06:32:49.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:32:49.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:32:49.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:32:49.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T06:32:49.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:32:49.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:32:49.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:32:49.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:32:49.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:32:49.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:32:49.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:32:50.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T06:32:50.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:32:50.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:32:50.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:32:50.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T06:32:50.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:32:50.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:32:50.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 5 = 1 ']' 2024-02-05T06:32:50.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:32:50.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 5 = 6 ']' 2024-02-05T06:32:50.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:32:50.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-6 1 1 2024-02-05T06:32:50.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-6 1 1 2024-02-05T06:32:50.966 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'psr_pool-6' created 2024-02-05T06:32:50.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:32:51.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:32:51.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:32:51.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:32:51.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:32:51.989 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:32:51.990 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:32:51.990 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:32:51.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:32:51.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:32:51.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:32:52.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:32:52.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:32:52.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:32:52.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:32:52.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:32:52.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:32:52.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:32:52.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:32:52.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:32:52.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:32:52.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-02-05T06:32:52.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-02-05T06:32:52.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-02-05T06:32:52.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:32:52.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-02-05T06:32:52.619 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:32:52.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:32:52.621 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-02-05T06:32:52.621 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:32:52.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-02-05T06:32:52.623 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836497 2024-02-05T06:32:52.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-02-05T06:32:52.623 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:52.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-02-05T06:32:52.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:32:53.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:32:53.927 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:54.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-02-05T06:32:54.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:32:54.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:32:54.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:32:54.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:32:54.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:32:54.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:32:54.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:32:54.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:32:54.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:32:54.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:32:54.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:32:54.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:32:54.929 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:32:54.929 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:32:54.929 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:32:55.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:32:55.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:32:55.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:32:55.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 6 = 1 ']' 2024-02-05T06:32:55.290 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:32:55.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 6 = 6 ']' 2024-02-05T06:32:55.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5868: TEST_scrub_warning: ceph osd pool set psr_pool-6 deep_scrub_interval 1209600 2024-02-05T06:32:55.734 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 6 deep_scrub_interval to 1209600 2024-02-05T06:32:55.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:32:55.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-7 1 1 2024-02-05T06:32:55.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-7 1 1 2024-02-05T06:32:56.270 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'psr_pool-7' created 2024-02-05T06:32:56.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:32:57.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:32:57.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:32:57.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:32:57.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:32:57.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:32:57.296 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:32:57.296 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:32:57.296 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:32:57.297 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:32:57.297 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:32:57.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:32:57.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:32:57.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:32:57.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:32:57.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:32:57.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:32:57.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:32:57.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:32:57.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:32:57.751 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:32:57.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-02-05T06:32:57.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-02-05T06:32:57.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-02-05T06:32:57.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:32:57.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-02-05T06:32:57.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:32:57.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:32:57.914 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-02-05T06:32:57.914 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:32:57.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-02-05T06:32:57.915 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836499 2024-02-05T06:32:57.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-02-05T06:32:57.916 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:58.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-02-05T06:32:58.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:32:59.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:32:59.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:32:59.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-02-05T06:32:59.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:32:59.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:32:59.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:32:59.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 7 == 0 2024-02-05T06:32:59.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:32:59.922 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:32:59.922 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:32:59.922 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:32:59.922 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:32:59.923 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:32:59.923 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:33:00.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-02-05T06:33:00.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:33:00.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:33:00.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:33:00.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 7 2024-02-05T06:33:00.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:33:00.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:33:00.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 7 = 1 ']' 2024-02-05T06:33:00.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:33:00.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 7 = 6 ']' 2024-02-05T06:33:00.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:33:00.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-8 1 1 2024-02-05T06:33:00.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-8 1 1 2024-02-05T06:33:01.213 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'psr_pool-8' created 2024-02-05T06:33:01.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:33:02.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:33:02.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:33:02.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:33:02.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:33:02.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:33:02.241 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:33:02.241 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:33:02.241 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:33:02.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:33:02.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:33:02.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:33:02.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:33:02.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:33:02.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:33:02.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:33:02.380 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:33:02.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:33:02.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:33:02.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:33:02.694 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:33:02.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-02-05T06:33:02.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-02-05T06:33:02.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-02-05T06:33:02.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:33:02.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-02-05T06:33:02.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:33:02.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:33:02.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-02-05T06:33:02.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:33:02.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-02-05T06:33:02.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-02-05T06:33:02.846 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836501 2024-02-05T06:33:02.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:33:03.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-02-05T06:33:03.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:33:04.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:33:04.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:33:04.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-02-05T06:33:04.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:33:04.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:33:04.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:33:04.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-02-05T06:33:04.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:33:04.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:33:04.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:33:04.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:33:04.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:33:04.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:33:04.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:33:05.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-02-05T06:33:05.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:33:05.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:33:05.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:33:05.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-02-05T06:33:05.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:33:05.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:33:05.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 8 = 1 ']' 2024-02-05T06:33:05.516 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:33:05.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 8 = 6 ']' 2024-02-05T06:33:05.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:33:05.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-9 1 1 2024-02-05T06:33:05.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-9 1 1 2024-02-05T06:33:05.957 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'psr_pool-9' created 2024-02-05T06:33:05.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:33:06.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:33:06.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:33:06.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:33:06.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:33:06.981 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:33:06.982 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:33:06.982 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:33:06.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:33:06.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:33:06.983 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:33:07.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:33:07.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:33:07.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:33:07.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:33:07.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:33:07.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:33:07.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:33:07.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:33:07.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:33:07.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:33:07.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-02-05T06:33:07.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-02-05T06:33:07.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-02-05T06:33:07.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:33:07.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-02-05T06:33:07.637 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:33:07.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:33:07.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-02-05T06:33:07.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:33:07.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-02-05T06:33:07.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-02-05T06:33:07.640 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836503 2024-02-05T06:33:07.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:33:07.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-02-05T06:33:07.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:33:08.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:33:08.941 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:33:09.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-02-05T06:33:09.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:33:10.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:33:10.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:33:10.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-02-05T06:33:10.574 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:33:10.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:33:10.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:33:10.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-02-05T06:33:10.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:33:10.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:33:10.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:33:10.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:33:10.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:33:10.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:33:10.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:33:11.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-02-05T06:33:11.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:33:11.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:33:11.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:33:11.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-02-05T06:33:11.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:33:11.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:33:11.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 9 = 1 ']' 2024-02-05T06:33:11.662 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:33:11.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 9 = 6 ']' 2024-02-05T06:33:11.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:33:11.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-10 1 1 2024-02-05T06:33:11.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-10 1 1 2024-02-05T06:33:12.144 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'psr_pool-10' created 2024-02-05T06:33:12.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:33:13.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:33:13.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:33:13.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:33:13.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:33:13.166 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:33:13.168 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:33:13.168 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:33:13.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:33:13.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:33:13.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:33:13.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:33:13.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:33:13.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:33:13.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:33:13.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:33:13.307 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:33:13.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:33:13.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:33:13.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:33:13.624 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:33:13.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-02-05T06:33:13.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-02-05T06:33:13.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-02-05T06:33:13.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:33:13.786 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-02-05T06:33:13.786 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:33:13.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:33:13.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-02-05T06:33:13.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:33:13.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-02-05T06:33:13.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-02-05T06:33:13.791 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836505 2024-02-05T06:33:13.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:33:14.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-02-05T06:33:14.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:33:15.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:33:15.122 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:33:15.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-02-05T06:33:15.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:33:15.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:33:15.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:33:15.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 == 0 2024-02-05T06:33:15.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:33:15.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:33:15.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:33:15.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:33:15.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:33:15.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:33:15.855 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:33:16.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=10 2024-02-05T06:33:16.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:33:16.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:33:16.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:33:16.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 10 = 10 2024-02-05T06:33:16.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:33:16.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:33:16.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 10 = 1 ']' 2024-02-05T06:33:16.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:33:16.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 10 = 6 ']' 2024-02-05T06:33:16.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5873: TEST_scrub_warning: local primary=0 2024-02-05T06:33:16.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: ceph osd set noscrub 2024-02-05T06:33:16.979 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:33:16.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5876: TEST_scrub_warning: ceph osd set nodeep-scrub 2024-02-05T06:33:17.448 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:33:17.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5877: TEST_scrub_warning: ceph config set global osd_scrub_interval_randomize_ratio 0 2024-02-05T06:33:17.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: ceph config set global osd_deep_scrub_randomize_ratio 0 2024-02-05T06:33:18.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: ceph config set global osd_scrub_max_interval 604800 2024-02-05T06:33:18.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5880: TEST_scrub_warning: ceph config set global osd_deep_scrub_interval 604800 2024-02-05T06:33:18.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: seq 1 5 2024-02-05T06:33:18.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-02-05T06:33:18.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 1 = 1 ']' 2024-02-05T06:33:18.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: overdue_seconds=1900800 2024-02-05T06:33:18.989 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 1900800 + 100 2024-02-05T06:33:18.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 1.0 schedule-scrub 1900900 2024-02-05T06:33:19.127 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:33:19.127 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:33:19.127 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:33:19.128 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-14T06:29:59.129488+0000" 2024-02-05T06:33:19.128 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:33:19.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-02-05T06:33:19.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 2 = 1 ']' 2024-02-05T06:33:19.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5889: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:33:19.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 993600 + 200 2024-02-05T06:33:19.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 2.0 schedule-scrub 993800 2024-02-05T06:33:19.292 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:33:19.292 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:33:19.292 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:33:19.292 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-24T18:28:19.294028+0000" 2024-02-05T06:33:19.292 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:33:19.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-02-05T06:33:19.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 3 = 1 ']' 2024-02-05T06:33:19.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5889: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:33:19.308 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 993600 + 300 2024-02-05T06:33:19.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 3.0 schedule-scrub 993900 2024-02-05T06:33:19.437 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:33:19.437 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:33:19.437 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:33:19.437 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-24T18:26:39.439012+0000" 2024-02-05T06:33:19.437 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:33:19.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-02-05T06:33:19.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 4 = 1 ']' 2024-02-05T06:33:19.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5889: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:33:19.452 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 993600 + 400 2024-02-05T06:33:19.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 4.0 schedule-scrub 994000 2024-02-05T06:33:19.583 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:33:19.583 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:33:19.583 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:33:19.583 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-24T18:24:59.585097+0000" 2024-02-05T06:33:19.584 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:33:19.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-02-05T06:33:19.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 5 = 1 ']' 2024-02-05T06:33:19.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5889: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:33:19.598 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 993600 + 500 2024-02-05T06:33:19.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 5.0 schedule-scrub 994100 2024-02-05T06:33:19.730 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:33:19.730 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:33:19.730 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:33:19.730 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-24T18:23:19.731659+0000" 2024-02-05T06:33:19.730 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:33:19.745 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:33:19.746 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 5 + 5 2024-02-05T06:33:19.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: seq 6 10 2024-02-05T06:33:19.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:33:19.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:33:19.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 6 = 6 ']' 2024-02-05T06:33:19.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5898: TEST_scrub_warning: overdue_seconds=1900800 2024-02-05T06:33:19.750 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 1900800 + 600 2024-02-05T06:33:19.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 6.0 schedule-deep-scrub 1901400 2024-02-05T06:33:19.878 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:33:19.878 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": true, 2024-02-05T06:33:19.879 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:33:19.879 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-14T06:21:39.880494+0000" 2024-02-05T06:33:19.879 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:33:19.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:33:19.893 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:33:19.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 7 = 6 ']' 2024-02-05T06:33:19.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:33:19.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 993600 + 700 2024-02-05T06:33:19.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 7.0 schedule-deep-scrub 994300 2024-02-05T06:33:20.026 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:33:20.026 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": true, 2024-02-05T06:33:20.026 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:33:20.026 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-24T18:20:00.027624+0000" 2024-02-05T06:33:20.026 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:33:20.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:33:20.040 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:33:20.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 8 = 6 ']' 2024-02-05T06:33:20.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:33:20.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 993600 + 800 2024-02-05T06:33:20.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 8.0 schedule-deep-scrub 994400 2024-02-05T06:33:20.166 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:33:20.166 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": true, 2024-02-05T06:33:20.166 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:33:20.166 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-24T18:18:20.168000+0000" 2024-02-05T06:33:20.167 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:33:20.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:33:20.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:33:20.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 9 = 6 ']' 2024-02-05T06:33:20.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:33:20.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 993600 + 900 2024-02-05T06:33:20.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 9.0 schedule-deep-scrub 994500 2024-02-05T06:33:20.303 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:33:20.303 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": true, 2024-02-05T06:33:20.303 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:33:20.303 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-24T18:16:40.304966+0000" 2024-02-05T06:33:20.303 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:33:20.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:33:20.316 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:33:20.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 10 = 6 ']' 2024-02-05T06:33:20.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:33:20.317 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 993600 + 1000 2024-02-05T06:33:20.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 10.0 schedule-deep-scrub 994600 2024-02-05T06:33:20.441 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:33:20.441 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": true, 2024-02-05T06:33:20.441 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:33:20.441 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-24T18:15:00.442955+0000" 2024-02-05T06:33:20.441 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:33:20.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5904: TEST_scrub_warning: flush_pg_stats 2024-02-05T06:33:20.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:33:20.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:33:20.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:33:20.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:33:20.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:33:20.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:33:20.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-02-05T06:33:20.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-02-05T06:33:20.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-02-05T06:33:20.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:33:20.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:33:20.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-02-05T06:33:20.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:33:20.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-02-05T06:33:20.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:33:20.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-02-05T06:33:20.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-02-05T06:33:20.911 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836508 2024-02-05T06:33:20.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:33:21.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836508 2024-02-05T06:33:21.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:33:22.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:33:22.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:33:22.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-02-05T06:33:22.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5906: TEST_scrub_warning: ceph health 2024-02-05T06:33:22.863 INFO:tasks.workunit.client.0.smithi148.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 10 pgs not scrubbed in time; 10 pool(s) do not have an application enabled; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2024-02-05T06:33:22.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5907: TEST_scrub_warning: ceph health detail 2024-02-05T06:33:23.258 INFO:tasks.workunit.client.0.smithi148.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 10 pgs not scrubbed in time; 10 pool(s) do not have an application enabled; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2024-02-05T06:33:23.258 INFO:tasks.workunit.client.0.smithi148.stdout:[WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T06:33:23.258 INFO:tasks.workunit.client.0.smithi148.stdout:[WRN] PG_NOT_DEEP_SCRUBBED: 5 pgs not deep-scrubbed in time 2024-02-05T06:33:23.258 INFO:tasks.workunit.client.0.smithi148.stdout: pg 10.0 not deep-scrubbed since 2024-01-24T18:15:00.442955+0000 2024-02-05T06:33:23.258 INFO:tasks.workunit.client.0.smithi148.stdout: pg 9.0 not deep-scrubbed since 2024-01-24T18:16:40.304966+0000 2024-02-05T06:33:23.258 INFO:tasks.workunit.client.0.smithi148.stdout: pg 8.0 not deep-scrubbed since 2024-01-24T18:18:20.168000+0000 2024-02-05T06:33:23.258 INFO:tasks.workunit.client.0.smithi148.stdout: pg 7.0 not deep-scrubbed since 2024-01-24T18:20:00.027624+0000 2024-02-05T06:33:23.258 INFO:tasks.workunit.client.0.smithi148.stdout: pg 6.0 not deep-scrubbed since 2024-01-14T06:21:39.880494+0000 2024-02-05T06:33:23.258 INFO:tasks.workunit.client.0.smithi148.stdout:[WRN] PG_NOT_SCRUBBED: 10 pgs not scrubbed in time 2024-02-05T06:33:23.258 INFO:tasks.workunit.client.0.smithi148.stdout: pg 10.0 not scrubbed since 2024-01-24T18:15:00.442955+0000 2024-02-05T06:33:23.259 INFO:tasks.workunit.client.0.smithi148.stdout: pg 9.0 not scrubbed since 2024-01-24T18:16:40.304966+0000 2024-02-05T06:33:23.259 INFO:tasks.workunit.client.0.smithi148.stdout: pg 8.0 not scrubbed since 2024-01-24T18:18:20.168000+0000 2024-02-05T06:33:23.259 INFO:tasks.workunit.client.0.smithi148.stdout: pg 7.0 not scrubbed since 2024-01-24T18:20:00.027624+0000 2024-02-05T06:33:23.259 INFO:tasks.workunit.client.0.smithi148.stdout: pg 6.0 not scrubbed since 2024-01-14T06:21:39.880494+0000 2024-02-05T06:33:23.259 INFO:tasks.workunit.client.0.smithi148.stdout: pg 5.0 not scrubbed since 2024-01-24T18:23:19.731659+0000 2024-02-05T06:33:23.259 INFO:tasks.workunit.client.0.smithi148.stdout: pg 4.0 not scrubbed since 2024-01-24T18:24:59.585097+0000 2024-02-05T06:33:23.259 INFO:tasks.workunit.client.0.smithi148.stdout: pg 3.0 not scrubbed since 2024-01-24T18:26:39.439012+0000 2024-02-05T06:33:23.259 INFO:tasks.workunit.client.0.smithi148.stdout: pg 2.0 not scrubbed since 2024-01-24T18:28:19.294028+0000 2024-02-05T06:33:23.259 INFO:tasks.workunit.client.0.smithi148.stdout: pg 1.0 not scrubbed since 2024-01-14T06:29:59.129488+0000 2024-02-05T06:33:23.259 INFO:tasks.workunit.client.0.smithi148.stdout:[WRN] POOL_APP_NOT_ENABLED: 10 pool(s) do not have an application enabled 2024-02-05T06:33:23.259 INFO:tasks.workunit.client.0.smithi148.stdout: application not enabled on pool 'psr_pool-1' 2024-02-05T06:33:23.259 INFO:tasks.workunit.client.0.smithi148.stdout: application not enabled on pool 'psr_pool-2' 2024-02-05T06:33:23.259 INFO:tasks.workunit.client.0.smithi148.stdout: application not enabled on pool 'psr_pool-3' 2024-02-05T06:33:23.260 INFO:tasks.workunit.client.0.smithi148.stdout: application not enabled on pool 'psr_pool-4' 2024-02-05T06:33:23.260 INFO:tasks.workunit.client.0.smithi148.stdout: application not enabled on pool 'psr_pool-5' 2024-02-05T06:33:23.260 INFO:tasks.workunit.client.0.smithi148.stdout: application not enabled on pool 'psr_pool-6' 2024-02-05T06:33:23.260 INFO:tasks.workunit.client.0.smithi148.stdout: application not enabled on pool 'psr_pool-7' 2024-02-05T06:33:23.260 INFO:tasks.workunit.client.0.smithi148.stdout: application not enabled on pool 'psr_pool-8' 2024-02-05T06:33:23.260 INFO:tasks.workunit.client.0.smithi148.stdout: application not enabled on pool 'psr_pool-9' 2024-02-05T06:33:23.260 INFO:tasks.workunit.client.0.smithi148.stdout: application not enabled on pool 'psr_pool-10' 2024-02-05T06:33:23.260 INFO:tasks.workunit.client.0.smithi148.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-05T06:33:23.260 INFO:tasks.workunit.client.0.smithi148.stdout:[WRN] POOL_NO_REDUNDANCY: 10 pool(s) have no replicas configured 2024-02-05T06:33:23.260 INFO:tasks.workunit.client.0.smithi148.stdout: pool 'psr_pool-1' has no replicas configured 2024-02-05T06:33:23.260 INFO:tasks.workunit.client.0.smithi148.stdout: pool 'psr_pool-2' has no replicas configured 2024-02-05T06:33:23.260 INFO:tasks.workunit.client.0.smithi148.stdout: pool 'psr_pool-3' has no replicas configured 2024-02-05T06:33:23.260 INFO:tasks.workunit.client.0.smithi148.stdout: pool 'psr_pool-4' has no replicas configured 2024-02-05T06:33:23.261 INFO:tasks.workunit.client.0.smithi148.stdout: pool 'psr_pool-5' has no replicas configured 2024-02-05T06:33:23.261 INFO:tasks.workunit.client.0.smithi148.stdout: pool 'psr_pool-6' has no replicas configured 2024-02-05T06:33:23.261 INFO:tasks.workunit.client.0.smithi148.stdout: pool 'psr_pool-7' has no replicas configured 2024-02-05T06:33:23.261 INFO:tasks.workunit.client.0.smithi148.stdout: pool 'psr_pool-8' has no replicas configured 2024-02-05T06:33:23.261 INFO:tasks.workunit.client.0.smithi148.stdout: pool 'psr_pool-9' has no replicas configured 2024-02-05T06:33:23.261 INFO:tasks.workunit.client.0.smithi148.stdout: pool 'psr_pool-10' has no replicas configured 2024-02-05T06:33:23.261 INFO:tasks.workunit.client.0.smithi148.stdout:[WRN] TOO_FEW_OSDS: OSD count 1 < osd_pool_default_size 3 2024-02-05T06:33:23.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: ceph health 2024-02-05T06:33:23.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: grep -q ' pgs not deep-scrubbed in time' 2024-02-05T06:33:23.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5909: TEST_scrub_warning: ceph health 2024-02-05T06:33:23.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5909: TEST_scrub_warning: grep -q ' pgs not scrubbed in time' 2024-02-05T06:33:24.002 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5914: TEST_scrub_warning: wc -l 2024-02-05T06:33:24.002 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5914: TEST_scrub_warning: ceph health detail 2024-02-05T06:33:24.003 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5914: TEST_scrub_warning: grep 'not scrubbed since' 2024-02-05T06:33:24.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5914: TEST_scrub_warning: COUNT=10 2024-02-05T06:33:24.402 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5916: TEST_scrub_warning: expr 5+5 2024-02-05T06:33:24.403 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5916: TEST_scrub_warning: (( 10 != 5 && 10 != 5+5 )) 2024-02-05T06:33:24.407 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5920: TEST_scrub_warning: ceph health detail 2024-02-05T06:33:24.407 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5920: TEST_scrub_warning: grep 'not deep-scrubbed since' 2024-02-05T06:33:24.407 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5920: TEST_scrub_warning: wc -l 2024-02-05T06:33:24.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5920: TEST_scrub_warning: COUNT=5 2024-02-05T06:33:24.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5921: TEST_scrub_warning: '[' 5 '!=' 5 ']' 2024-02-05T06:33:24.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:33:24.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:33:24.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:33:24.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:33:24.920 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:33:24.920 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:33:24.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:33:24.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:33:24.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:33:25.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:33:25.232 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:33:25.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:33:25.234 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:33:25.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:33:25.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:33:25.236 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:33:25.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:33:25.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:33:25.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:33:25.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:33:25.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:33:25.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:33:25.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:33:25.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:33:25.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:33:25.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:33:25.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:33:25.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:33:25.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:33:25.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:33:25.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:33:25.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:33:25.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:33:25.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:33:25.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:33:25.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:33:25.258 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:33:25.258 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:33:25.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:33:25.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:33:25.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:33:25.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:33:25.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:33:25.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:33:25.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:33:25.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:33:25.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:33:25.265 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:33:25.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:33:25.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:33:25.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:33:25.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:33:25.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:33:25.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:33:25.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:33:25.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:33:25.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:33:25.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:33:25.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:33:25.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:33:25.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:33:25.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:33:25.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:33:25.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:33:25.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:33:25.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T06:33:25.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:33:25.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:33:25.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:33:25.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:33:25.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_skip_non_repair_during_recovery td/osd-scrub-repair 2024-02-05T06:33:25.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:136: TEST_skip_non_repair_during_recovery: local dir=td/osd-scrub-repair 2024-02-05T06:33:25.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:137: TEST_skip_non_repair_during_recovery: local poolname=rbd 2024-02-05T06:33:25.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:139: TEST_skip_non_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-02-05T06:33:25.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:33:25.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:33:25.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:33:25.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:33:25.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:33:25.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-02-05T06:33:25.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:33:25.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:33:25.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:33:25.671 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:33:25.671 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:33:25.671 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:33:25.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:33:25.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T06:33:25.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:33:25.892 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:33:25.892 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:33:25.892 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:33:25.892 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:33:25.893 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:33:25.893 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:33:25.893 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:33:25.893 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:33:25.894 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:33:25.894 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:33:25.894 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:33:25.894 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:33:25.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:33:25.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T06:33:26.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:33:26.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:33:26.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:33:26.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:33:26.011 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:33:26.013 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:33:26.013 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:33:26.013 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:33:26.013 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:33:26.013 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:33:26.013 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:33:26.013 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:33:26.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:33:26.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T06:33:26.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:140: TEST_skip_non_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-02-05T06:33:26.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:33:26.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:33:26.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:33:26.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:33:26.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:33:26.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:33:26.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:33:26.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:33:26.991 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:33:26.992 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:33:26.992 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:33:26.992 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:33:26.992 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:33:26.993 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:33:26.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:33:27.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:141: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T06:33:27.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:33:27.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:33:27.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:33:27.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:33:27.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:33:27.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:33:27.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:33:27.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:33:27.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:33:27.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:33:27.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:33:27.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:33:27.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:33:27.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:33:27.031 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:33:27.031 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:33:27.031 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:33:27.031 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:33:27.031 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:33:27.031 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:33:27.031 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:33:27.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:33:27.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:33:27.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:33:27.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:33:27.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:33:27.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:33:27.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:33:27.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:33:27.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:33:27.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:33:27.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:33:27.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-02-05T06:33:27.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:33:27.034 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:33:27.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=959e7540-4fb3-4ceb-8308-c2af1eb3268f 2024-02-05T06:33:27.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 959e7540-4fb3-4ceb-8308-c2af1eb3268f' 2024-02-05T06:33:27.034 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 959e7540-4fb3-4ceb-8308-c2af1eb3268f 2024-02-05T06:33:27.034 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:33:27.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3gcBliRYhAxAAhyxxg8FB2Q+5kvA9I26axg== 2024-02-05T06:33:27.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3gcBliRYhAxAAhyxxg8FB2Q+5kvA9I26axg=="}' 2024-02-05T06:33:27.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 959e7540-4fb3-4ceb-8308-c2af1eb3268f -i td/osd-scrub-repair/0/new.json 2024-02-05T06:33:27.344 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:33:27.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:33:27.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQA3gcBliRYhAxAAhyxxg8FB2Q+5kvA9I26axg== --osd-uuid 959e7540-4fb3-4ceb-8308-c2af1eb3268f 2024-02-05T06:33:27.400 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:33:27.399+0000 7f982d6ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:33:27.411 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:33:27.407+0000 7f982d6ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:33:27.413 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:33:27.411+0000 7f982d6ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:33:27.413 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:33:27.411+0000 7f982d6ce640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:33:27.413 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:33:27.411+0000 7f982d6ce640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:33:27.413 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:33:27.411+0000 7f982d6ce640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:33:27.512 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:33:27.507+0000 7f982d6ce640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:33:33.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:33:33.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:33:33.600 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:33:33.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:33:33.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:33:34.026 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:33:34.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:33:34.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:33:34.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T06:33:34.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:33:34.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:33:34.061 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:33:34.059+0000 7f27caad6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:33:34.095 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:33:34.091+0000 7f27caad6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:33:34.097 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:33:34.095+0000 7f27caad6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:33:34.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:33:34.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:33:34.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:33:34.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:33:34.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:33:34.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:34.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:33:34.312 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:33:34.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:34.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:34.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:34.691 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:33:34.687+0000 7f27caad6640 -1 Falling back to public interface 2024-02-05T06:33:35.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:35.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:35.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:33:35.637 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:33:35.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:35.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:35.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:36.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:36.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:36.965 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:33:36.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:33:36.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:36.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:37.241 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:33:37.239+0000 7f27caad6640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:33:37.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:38.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:38.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:38.272 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:33:38.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:33:38.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:38.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:38.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:39.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:39.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:39.584 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:33:39.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:33:39.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:39.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:39.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:40.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:40.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:40.989 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:33:40.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:33:40.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:40.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:41.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:42.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:42.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:42.304 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:33:42.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:33:42.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:42.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:42.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:43.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:43.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:43.621 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:33:43.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:33:43.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:43.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:43.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:44.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:44.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:44.989 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:33:44.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:33:44.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:44.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:45.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:46.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:46.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:46.306 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:33:46.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:33:46.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:46.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:46.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:47.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:47.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:47.624 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:33:47.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:33:47.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:47.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:47.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:48.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:48.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:48.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:33:48.991 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:33:48.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:48.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:49.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:50.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:50.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:50.313 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:33:50.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:33:50.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:50.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:50.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:51.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:51.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:51.955 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:33:51.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:33:51.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:51.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:52.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:53.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:53.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:53.681 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:33:53.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:33:53.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:53.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:54.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:55.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:55.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:55.494 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:33:55.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:33:55.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:55.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:56.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:57.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:57.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:57.014 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:33:57.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:33:57.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:57.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:57.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:33:58.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:33:58.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:33:58.501 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:33:58.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:33:58.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:33:58.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:33:59.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:00.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:00.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:00.274 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:34:00.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:34:00.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:00.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:34:00.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:01.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:01.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:01.623 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:34:01.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:34:01.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:01.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:34:01.937 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2944423077,v1:127.0.0.1:6803/2944423077] [v2:127.0.0.1:6804/2944423077,v1:127.0.0.1:6805/2944423077] exists,up 959e7540-4fb3-4ceb-8308-c2af1eb3268f 2024-02-05T06:34:01.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:34:01.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:34:01.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:34:01.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:144: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T06:34:01.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:34:01.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:34:01.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:34:01.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:34:01.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:34:01.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:34:01.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:34:01.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:34:01.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:34:01.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:34:01.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:34:01.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:34:01.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:34:01.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:34:01.941 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:34:01.941 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:34:01.941 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:34:01.941 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:34:01.941 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:34:01.941 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:34:01.943 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:34:01.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:34:01.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:34:01.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:34:01.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:34:01.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:34:01.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:34:01.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:34:01.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:34:01.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:34:01.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:34:01.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:34:01.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-02-05T06:34:01.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:34:01.944 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:34:01.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62df3e68-86be-401f-83f9-46f0eafbdce6 2024-02-05T06:34:01.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 62df3e68-86be-401f-83f9-46f0eafbdce6' 2024-02-05T06:34:01.945 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 62df3e68-86be-401f-83f9-46f0eafbdce6 2024-02-05T06:34:01.945 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:34:01.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZgcBlRJCQORAAK7VhG/otiZqufp9WcgmXyw== 2024-02-05T06:34:01.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZgcBlRJCQORAAK7VhG/otiZqufp9WcgmXyw=="}' 2024-02-05T06:34:01.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62df3e68-86be-401f-83f9-46f0eafbdce6 -i td/osd-scrub-repair/1/new.json 2024-02-05T06:34:02.313 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:34:02.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T06:34:02.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBZgcBlRJCQORAAK7VhG/otiZqufp9WcgmXyw== --osd-uuid 62df3e68-86be-401f-83f9-46f0eafbdce6 2024-02-05T06:34:02.363 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:02.358+0000 7fa7103e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:34:02.407 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:02.402+0000 7fa7103e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:34:02.409 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:02.406+0000 7fa7103e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:34:02.409 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:02.406+0000 7fa7103e4640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:34:02.409 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:02.406+0000 7fa7103e4640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:34:02.409 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:02.406+0000 7fa7103e4640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:34:02.482 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:02.482+0000 7fa7103e4640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T06:34:08.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T06:34:08.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:34:08.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:34:08.708 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:34:08.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:34:09.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:34:09.089 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:34:09.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T06:34:09.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:34:09.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:34:09.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:34:09.118 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:09.114+0000 7f6543edf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:34:09.121 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:09.118+0000 7f6543edf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:34:09.122 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:09.118+0000 7f6543edf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:34:09.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:34:09.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:34:09.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:34:09.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:34:09.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:34:09.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:09.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:34:09.432 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:34:09.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:09.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:09.655 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:09.650+0000 7f6543edf640 -1 Falling back to public interface 2024-02-05T06:34:09.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:10.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:10.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:10.745 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:34:10.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:34:10.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:10.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:11.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:12.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:12.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:12.054 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:34:12.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:34:12.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:12.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:12.365 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:12.366+0000 7f6543edf640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:34:12.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:13.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:13.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:13.381 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:34:13.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:34:13.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:13.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:13.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:14.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:14.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:14.695 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:34:14.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:34:14.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:14.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:15.016 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:16.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:16.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:16.019 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:34:16.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:34:16.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:16.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:16.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:17.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:17.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:17.615 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:34:17.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:34:17.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:17.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:17.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:18.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:18.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:18.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:34:18.937 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:34:18.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:18.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:19.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:20.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:20.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:20.250 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:34:20.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:34:20.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:20.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:20.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:21.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:21.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:21.567 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:34:21.567 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:34:21.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:21.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:21.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:22.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:22.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:22.978 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:34:22.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:34:22.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:22.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:23.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:24.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:24.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:24.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:34:24.430 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:34:24.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:24.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:24.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:25.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:25.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:25.808 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:34:25.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:34:25.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:25.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:26.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:27.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:27.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:27.121 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:34:27.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:34:27.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:27.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:27.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:28.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:28.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:28.881 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:34:28.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:34:28.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:28.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:29.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:30.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:30.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:30.338 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:34:30.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:34:30.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:30.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:31.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:32.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:32.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:32.141 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:34:32.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:34:32.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:32.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:33.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:34.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:34.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:34.122 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:34:34.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:34:34.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:34.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:34.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:35.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:35.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:35.536 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:34:35.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:34:35.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:35.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:36.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:37.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:37.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:37.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:34:37.428 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:34:37.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:37.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:34:37.749 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4024546117,v1:127.0.0.1:6811/4024546117] [v2:127.0.0.1:6812/4024546117,v1:127.0.0.1:6813/4024546117] exists,up 62df3e68-86be-401f-83f9-46f0eafbdce6 2024-02-05T06:34:37.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:34:37.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:34:37.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:34:37.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:147: TEST_skip_non_repair_during_recovery: create_rbd_pool 2024-02-05T06:34:37.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T06:34:38.071 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' does not exist 2024-02-05T06:34:38.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T06:34:38.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T06:34:38.513 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'rbd' created 2024-02-05T06:34:38.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:34:39.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T06:34:41.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:148: TEST_skip_non_repair_during_recovery: wait_for_clean 2024-02-05T06:34:41.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:34:41.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:34:41.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:34:41.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:34:41.471 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:34:41.471 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:34:41.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:34:41.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:34:41.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:34:41.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:34:41.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:34:41.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:34:41.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:34:41.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:34:41.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:34:41.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:34:41.928 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:34:41.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:34:41.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:34:41.928 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:34:42.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-02-05T06:34:42.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-02-05T06:34:42.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-02-05T06:34:42.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:34:42.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:34:42.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-02-05T06:34:42.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-02-05T06:34:42.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672967' 2024-02-05T06:34:42.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:34:42.291 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-02-05T06:34:42.291 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:34:42.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:34:42.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-02-05T06:34:42.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:34:42.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-02-05T06:34:42.294 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836496 2024-02-05T06:34:42.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-02-05T06:34:42.295 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:34:42.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836496 2024-02-05T06:34:42.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:34:43.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:34:43.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:34:43.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-02-05T06:34:43.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:34:43.894 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-02-05T06:34:43.894 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:34:43.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:34:43.897 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-02-05T06:34:43.897 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:34:43.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-02-05T06:34:43.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-02-05T06:34:43.899 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672967 2024-02-05T06:34:43.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:34:44.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672967 2024-02-05T06:34:44.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:34:44.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:34:44.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:34:44.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T06:34:44.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:34:44.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:34:44.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:34:44.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:34:44.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:34:44.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:34:44.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:34:44.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:34:44.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:34:44.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:34:44.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:34:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T06:34:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:34:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:34:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:150: TEST_skip_non_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-02-05T06:34:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:34:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-02-05T06:34:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T06:34:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:34:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:34:45.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:34:45.636 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:34:45.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:34:46.207 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:34:46.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:34:46.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:34:46.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T06:34:46.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_skip_non_repair_during_recovery: get_not_primary rbd SOMETHING 2024-02-05T06:34:46.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-02-05T06:34:46.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T06:34:46.513 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-02-05T06:34:46.513 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-02-05T06:34:46.513 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T06:34:46.513 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-02-05T06:34:46.514 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:34:46.835 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T06:34:46.836 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T06:34:46.836 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-02-05T06:34:47.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_skip_non_repair_during_recovery: scrub_and_not_schedule td/osd-scrub-repair rbd 0 2024-02-05T06:34:47.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:155: scrub_and_not_schedule: local dir=td/osd-scrub-repair 2024-02-05T06:34:47.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:156: scrub_and_not_schedule: local poolname=rbd 2024-02-05T06:34:47.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:157: scrub_and_not_schedule: local osd=0 2024-02-05T06:34:47.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: get_pg rbd SOMETHING 2024-02-05T06:34:47.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-02-05T06:34:47.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T06:34:47.162 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-02-05T06:34:47.162 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:34:47.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: local pg=1.3 2024-02-05T06:34:47.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:163: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-02-05T06:34:47.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T06:34:47.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:34:47.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:34:47.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T06:34:47.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:163: scrub_and_not_schedule: local last_scrub=2024-02-05T06:34:38.441822+0000 2024-02-05T06:34:47.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:164: scrub_and_not_schedule: ceph tell 1.3 schedule-scrub 2024-02-05T06:34:47.995 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:34:47.995 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:34:47.996 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:34:47.996 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-29T06:33:07.996335+0000" 2024-02-05T06:34:47.996 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:34:48.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i=0 )) 2024-02-05T06:34:48.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-02-05T06:34:48.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-02-05T06:34:48.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T06:34:48.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:34:48.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:34:48.010 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T06:34:48.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-02-05T06:34:38.441822+0000 '>' 2024-02-05T06:34:38.441822+0000 2024-02-05T06:34:48.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-02-05T06:34:49.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-02-05T06:34:49.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-02-05T06:34:49.325 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-02-05T06:34:49.325 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T06:34:49.325 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:34:49.325 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:34:49.325 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T06:34:49.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-02-05T06:34:38.441822+0000 '>' 2024-02-05T06:34:38.441822+0000 2024-02-05T06:34:49.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-02-05T06:34:50.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-02-05T06:34:50.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-02-05T06:34:50.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-02-05T06:34:50.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T06:34:50.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:34:50.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:34:50.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T06:34:50.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-02-05T06:34:38.441822+0000 '>' 2024-02-05T06:34:38.441822+0000 2024-02-05T06:34:50.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-02-05T06:34:51.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-02-05T06:34:51.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-02-05T06:34:51.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:179: scrub_and_not_schedule: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T06:34:51.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:34:51.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:34:51.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:34:51.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:34:51.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T06:34:51.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:34:51.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:34:51.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:34:51.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:34:51.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:34:51.931 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:34:51.931 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:34:51.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:34:51.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:34:51.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:34:52.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:34:52.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T06:34:52.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:34:52.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:34:52.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:34:52.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:34:52.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:34:52.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-02-05T06:34:53.598 INFO:tasks.workunit.client.0.smithi148.stdout:_ 2024-02-05T06:34:53.598 INFO:tasks.workunit.client.0.smithi148.stdout:snapset 2024-02-05T06:34:53.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:34:53.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:34:53.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:34:53.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:34:53.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:34:53.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:34:53.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:34:53.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:34:53.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:34:53.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:34:53.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:34:53.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:34:53.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:34:53.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:34:53.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:34:53.904 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:34:53.904 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:34:53.904 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:34:53.906 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:34:53.906 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:34:53.906 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:34:53.906 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:34:53.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:34:53.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:34:53.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:34:53.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:34:53.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:34:53.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:34:53.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:34:53.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:34:53.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:34:53.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:34:53.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:34:53.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:34:53.907 INFO:tasks.workunit.client.0.smithi148.stderr:start osd.0 2024-02-05T06:34:53.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:34:53.908 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:34:53.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:34:53.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:34:53.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:34:53.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:34:53.941 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:53.937+0000 7f70de8b7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:34:53.943 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:53.941+0000 7f70de8b7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:34:53.944 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:53.941+0000 7f70de8b7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:34:54.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:34:54.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:34:54.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:34:54.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:34:54.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:34:54.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:54.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:34:54.228 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:34:54.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:34:54.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:54.480 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:54.477+0000 7f70de8b7640 -1 Falling back to public interface 2024-02-05T06:34:54.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:55.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:55.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:55.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:34:55.540 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:34:55.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:55.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:34:55.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:56.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:56.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:56.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:34:56.884 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:34:56.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:56.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:34:56.973 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:34:56.969+0000 7f70de8b7640 -1 osd.0 21 log_to_monitors true 2024-02-05T06:34:57.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:34:58.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:34:58.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:34:58.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:34:58.360 INFO:tasks.workunit.client.0.smithi148.stderr:3 2024-02-05T06:34:58.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:34:58.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:34:58.672 INFO:tasks.workunit.client.0.smithi148.stderr:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1622333229,v1:127.0.0.1:6803/1622333229] [v2:127.0.0.1:6804/1622333229,v1:127.0.0.1:6805/1622333229] exists,up 959e7540-4fb3-4ceb-8308-c2af1eb3268f 2024-02-05T06:34:58.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:34:58.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:34:58.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:34:58.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:34:58.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:34:58.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:34:58.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:34:58.673 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:34:58.673 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:34:58.673 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:34:58.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:34:58.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:34:58.674 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:34:58.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:34:58.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:34:58.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:34:58.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:34:58.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:34:58.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:34:59.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:34:59.097 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:34:59.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:34:59.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:34:59.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:34:59.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-02-05T06:34:59.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-02-05T06:34:59.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-02-05T06:34:59.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:34:59.283 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:34:59.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-02-05T06:34:59.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-02-05T06:34:59.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672972' 2024-02-05T06:34:59.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:34:59.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-02-05T06:34:59.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:34:59.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:34:59.428 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-02-05T06:34:59.428 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:34:59.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-02-05T06:34:59.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-02-05T06:34:59.429 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.0 seq 107374182402 2024-02-05T06:34:59.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:34:59.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-02-05T06:34:59.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:34:59.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-02-05T06:34:59.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:34:59.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:34:59.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-02-05T06:34:59.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:34:59.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-02-05T06:34:59.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-02-05T06:34:59.750 INFO:tasks.workunit.client.0.smithi148.stderr:waiting osd.1 seq 42949672972 2024-02-05T06:34:59.751 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:35:00.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-02-05T06:35:00.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:35:00.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:35:00.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:35:00.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T06:35:00.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:35:00.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:35:00.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:35:00.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:35:00.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:35:00.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:35:00.450 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:35:00.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:35:00.769 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:35:00.770 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:35:00.770 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:35:01.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T06:35:01.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:35:01.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:35:01.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:180: scrub_and_not_schedule: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T06:35:01.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:181: scrub_and_not_schedule: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T06:35:01.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:35:01.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:35:01.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:35:01.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:35:01.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:35:01.210 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:35:01.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:35:01.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:35:01.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:35:01.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:35:01.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:35:01.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:35:01.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:35:01.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:35:01.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:35:01.526 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:35:01.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:35:01.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:35:01.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:35:01.529 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:35:01.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:35:01.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:35:01.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:35:01.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:35:01.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:35:01.547 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:35:01.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:35:01.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:35:01.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:35:01.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:35:01.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:35:01.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:35:01.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:35:01.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:35:01.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:35:01.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:35:01.551 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:35:01.551 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:35:01.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:35:01.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:35:01.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:35:01.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:35:01.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:35:01.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:35:01.557 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:35:01.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:35:01.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:35:01.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:35:01.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:35:01.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:35:01.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:35:01.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:35:01.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:35:01.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:35:01.564 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:35:01.566 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:35:01.566 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:35:01.566 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:35:01.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:35:01.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:35:01.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:35:01.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:35:01.570 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:35:01.570 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:35:01.570 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:35:01.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T06:35:01.573 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:35:01.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:35:01.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:35:01.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:35:01.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_appends td/osd-scrub-repair 2024-02-05T06:35:01.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:903: TEST_unfound_erasure_coded_appends: unfound_erasure_coded td/osd-scrub-repair false 2024-02-05T06:35:01.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: unfound_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T06:35:01.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: unfound_erasure_coded: local allow_overwrites=false 2024-02-05T06:35:01.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:854: unfound_erasure_coded: local poolname=ecpool 2024-02-05T06:35:01.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:855: unfound_erasure_coded: local payload=ABCDEF 2024-02-05T06:35:01.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2024-02-05T06:35:01.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:35:01.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:35:01.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:35:01.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:35:01.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:35:01.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T06:35:01.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:35:01.960 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:35:01.960 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:35:01.960 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:35:01.960 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:35:01.961 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:35:01.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:35:01.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T06:35:02.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:35:02.173 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:35:02.173 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:35:02.173 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:35:02.173 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:35:02.173 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:35:02.174 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:35:02.174 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:35:02.174 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:35:02.175 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:35:02.175 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:35:02.175 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:35:02.175 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:35:02.175 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:35:02.176 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T06:35:02.292 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:35:02.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:35:02.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:35:02.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:35:02.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:35:02.293 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:35:02.293 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:35:02.293 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:35:02.294 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:35:02.294 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:35:02.294 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:35:02.294 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:35:02.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:35:02.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T06:35:02.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2024-02-05T06:35:02.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:35:02.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:35:02.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:35:02.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:35:02.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:35:02.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:35:03.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:35:03.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:35:03.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:35:03.271 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:35:03.271 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:35:03.271 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:35:03.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:35:03.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:35:03.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:35:03.308 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: seq 0 3 2024-02-05T06:35:03.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:35:03.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2024-02-05T06:35:03.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:35:03.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:35:03.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:35:03.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:35:03.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:35:03.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:35:03.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:35:03.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:35:03.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:35:03.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:35:03.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:35:03.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:35:03.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:35:03.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:35:03.312 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:35:03.312 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:35:03.312 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:35:03.312 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:35:03.312 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:35:03.312 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:35:03.313 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:35:03.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:35:03.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:35:03.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:35:03.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:35:03.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:35:03.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:35:03.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:35:03.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:35:03.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:35:03.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:35:03.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:35:03.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:35:03.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:35:03.316 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:35:03.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bb578e5b-a58d-4d6e-b434-b7a6746025fe 2024-02-05T06:35:03.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bb578e5b-a58d-4d6e-b434-b7a6746025fe' 2024-02-05T06:35:03.316 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 bb578e5b-a58d-4d6e-b434-b7a6746025fe 2024-02-05T06:35:03.317 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:35:03.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXgcBloIzLExAAnOhMSMhivZDIvHNEPx7b5A== 2024-02-05T06:35:03.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXgcBloIzLExAAnOhMSMhivZDIvHNEPx7b5A=="}' 2024-02-05T06:35:03.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bb578e5b-a58d-4d6e-b434-b7a6746025fe -i td/osd-scrub-repair/0/new.json 2024-02-05T06:35:03.591 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:35:03.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:35:03.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCXgcBloIzLExAAnOhMSMhivZDIvHNEPx7b5A== --osd-uuid bb578e5b-a58d-4d6e-b434-b7a6746025fe 2024-02-05T06:35:03.644 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:03.641+0000 7ffaa7e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:35:03.658 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:03.657+0000 7ffaa7e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:35:03.660 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:03.657+0000 7ffaa7e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:35:03.660 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:03.657+0000 7ffaa7e12640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:35:03.660 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:03.657+0000 7ffaa7e12640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:35:03.660 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:03.657+0000 7ffaa7e12640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:35:03.751 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:03.749+0000 7ffaa7e12640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:35:09.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:35:09.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:35:09.950 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:35:09.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:35:09.950 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:35:10.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:35:10.315 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:35:10.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:35:10.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:35:10.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:35:10.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:35:10.347 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:10.345+0000 7fb6f5905640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:35:10.349 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:10.345+0000 7fb6f5905640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:35:10.350 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:10.349+0000 7fb6f5905640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:35:10.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:35:10.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:35:10.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:35:10.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:35:10.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:35:10.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:10.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:35:10.627 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:35:10.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:10.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:10.896 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:10.893+0000 7fb6f5905640 -1 Falling back to public interface 2024-02-05T06:35:10.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:11.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:11.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:11.943 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:35:11.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:35:11.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:11.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:12.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:13.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:13.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:13.252 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:35:13.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:35:13.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:13.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:13.434 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:13.433+0000 7fb6f5905640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:35:13.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:14.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:14.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:14.558 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:35:14.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:35:14.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:14.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:14.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:15.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:15.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:15.878 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:35:15.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:35:15.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:15.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:16.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:17.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:17.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:17.194 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:35:17.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:35:17.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:17.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:17.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:18.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:18.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:18.500 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:35:18.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:35:18.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:18.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:18.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:19.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:19.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:19.822 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:35:19.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:35:19.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:19.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:20.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:21.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:21.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:21.183 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:35:21.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:35:21.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:21.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:21.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:22.509 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:35:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:35:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:22.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:22.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:23.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:23.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:23.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:35:23.822 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:35:23.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:23.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:24.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:25.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:25.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:25.198 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:35:25.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:35:25.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:25.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:25.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:26.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:26.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:26.509 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:35:26.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:35:26.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:26.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:26.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:27.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:27.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:27.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:35:27.916 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:35:27.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:27.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:28.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:29.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:29.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:29.221 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:35:29.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:35:29.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:29.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:29.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:30.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:30.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:30.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:35:30.710 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:35:30.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:30.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:31.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:32.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:32.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:32.508 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:35:32.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:35:32.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:32.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:33.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:34.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:34.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:34.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:35:34.311 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:35:34.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:34.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:34.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:35.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:35.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:35.788 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:35:35.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:35:35.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:35.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:36.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:37.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:37.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:37.467 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:35:37.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:35:37.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:37.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:35:37.822 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1752492560,v1:127.0.0.1:6803/1752492560] [v2:127.0.0.1:6804/1752492560,v1:127.0.0.1:6805/1752492560] exists,up bb578e5b-a58d-4d6e-b434-b7a6746025fe 2024-02-05T06:35:37.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:35:37.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:35:37.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:35:37.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:35:37.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2024-02-05T06:35:37.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:35:37.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:35:37.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:35:37.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:35:37.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:35:37.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:35:37.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:35:37.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:35:37.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:35:37.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:35:37.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:35:37.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:35:37.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:35:37.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:35:37.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:35:37.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:35:37.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:35:37.827 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:35:37.827 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:35:37.827 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:35:37.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:35:37.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:35:37.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:35:37.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:35:37.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:35:37.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:35:37.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:35:37.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:35:37.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:35:37.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:35:37.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:35:37.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:35:37.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:35:37.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:35:37.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:35:37.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a6df43bc-bcfa-495b-accd-bbaf1035810a 2024-02-05T06:35:37.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a6df43bc-bcfa-495b-accd-bbaf1035810a' 2024-02-05T06:35:37.830 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 a6df43bc-bcfa-495b-accd-bbaf1035810a 2024-02-05T06:35:37.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:35:37.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC5gcBlJppoMhAAtfa8U/3uEjH+qgGqOq7ONQ== 2024-02-05T06:35:37.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC5gcBlJppoMhAAtfa8U/3uEjH+qgGqOq7ONQ=="}' 2024-02-05T06:35:37.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a6df43bc-bcfa-495b-accd-bbaf1035810a -i td/osd-scrub-repair/1/new.json 2024-02-05T06:35:38.232 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:35:38.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T06:35:38.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC5gcBlJppoMhAAtfa8U/3uEjH+qgGqOq7ONQ== --osd-uuid a6df43bc-bcfa-495b-accd-bbaf1035810a 2024-02-05T06:35:38.288 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:38.284+0000 7ff2334ee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:35:38.325 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:38.320+0000 7ff2334ee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:35:38.326 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:38.324+0000 7ff2334ee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:35:38.327 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:38.324+0000 7ff2334ee640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:35:38.327 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:38.324+0000 7ff2334ee640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:35:38.327 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:38.324+0000 7ff2334ee640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:35:38.416 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:38.412+0000 7ff2334ee640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T06:35:44.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T06:35:44.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:35:44.783 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:35:44.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:35:44.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:35:45.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:35:45.159 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:35:45.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:35:45.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:35:45.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:35:45.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:35:45.192 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:45.188+0000 7f8c103e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:35:45.224 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:45.220+0000 7f8c103e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:35:45.226 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:45.224+0000 7f8c103e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:35:45.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:35:45.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:35:45.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:35:45.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:35:45.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:35:45.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:45.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:35:45.481 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:35:45.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:45.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:35:45.776 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:45.772+0000 7f8c103e0640 -1 Falling back to public interface 2024-02-05T06:35:45.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:46.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:46.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:46.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:35:46.812 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:35:46.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:46.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:35:47.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:48.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:48.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:48.129 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:35:48.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:35:48.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:48.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:35:48.349 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:35:48.344+0000 7f8c103e0640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:35:48.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:49.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:49.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:49.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:35:49.442 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:35:49.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:49.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:35:49.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:50.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:50.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:50.789 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:35:50.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:35:50.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:50.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:35:51.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:52.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:52.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:52.109 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:35:52.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:35:52.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:52.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:35:52.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:53.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:53.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:53.499 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:35:53.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:35:53.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:53.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:35:53.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:54.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:54.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:54.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:35:54.840 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:35:54.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:54.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:35:55.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:56.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:56.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:56.157 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:35:56.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:35:56.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:56.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:35:56.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:57.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:57.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:57.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:35:57.493 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:35:57.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:57.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:35:57.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:35:58.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:35:58.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:35:58.834 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:35:58.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:35:58.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:35:58.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:35:59.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:00.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:00.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:00.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:36:00.407 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:36:00.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:00.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:36:00.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:01.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:01.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:01.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:36:01.857 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:36:01.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:01.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:36:02.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:03.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:03.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:03.179 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:36:03.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:36:03.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:03.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:36:03.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:04.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:04.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:04.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:36:04.914 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:36:04.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:04.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:36:05.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:06.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:06.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:06.533 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:36:06.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:36:06.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:06.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:36:07.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:08.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:08.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:08.349 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:36:08.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:36:08.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:08.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:36:09.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:10.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:10.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:10.277 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:36:10.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:36:10.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:10.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:36:10.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:11.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:11.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:11.738 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:36:11.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:36:11.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:11.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:36:12.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:13.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:13.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:13.149 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:36:13.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:36:13.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:13.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:36:13.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:14.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:14.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:14.487 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:36:14.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:36:14.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:14.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:36:14.806 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/332275199,v1:127.0.0.1:6811/332275199] [v2:127.0.0.1:6812/332275199,v1:127.0.0.1:6813/332275199] exists,up a6df43bc-bcfa-495b-accd-bbaf1035810a 2024-02-05T06:36:14.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:36:14.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:36:14.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:36:14.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:36:14.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2024-02-05T06:36:14.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:36:14.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:36:14.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T06:36:14.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:36:14.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:36:14.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:36:14.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:36:14.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:36:14.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:36:14.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:36:14.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:36:14.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:36:14.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:36:14.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:36:14.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:36:14.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:36:14.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:36:14.810 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:36:14.810 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:36:14.811 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:36:14.811 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:36:14.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:36:14.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:36:14.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:36:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:36:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:36:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:36:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:36:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:36:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:36:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:36:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:36:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:36:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:36:14.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:36:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=553baed5-50fc-49ff-840d-ed74cfa37a16 2024-02-05T06:36:14.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 553baed5-50fc-49ff-840d-ed74cfa37a16' 2024-02-05T06:36:14.814 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 553baed5-50fc-49ff-840d-ed74cfa37a16 2024-02-05T06:36:14.814 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:36:14.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDegcBlp/WwMRAA0+hvYDJj4HltXcOIQKidWA== 2024-02-05T06:36:14.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDegcBlp/WwMRAA0+hvYDJj4HltXcOIQKidWA=="}' 2024-02-05T06:36:14.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 553baed5-50fc-49ff-840d-ed74cfa37a16 -i td/osd-scrub-repair/2/new.json 2024-02-05T06:36:15.218 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:36:15.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T06:36:15.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDegcBlp/WwMRAA0+hvYDJj4HltXcOIQKidWA== --osd-uuid 553baed5-50fc-49ff-840d-ed74cfa37a16 2024-02-05T06:36:15.274 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:15.267+0000 7f8098e51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:36:15.320 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:15.315+0000 7f8098e51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:36:15.322 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:15.315+0000 7f8098e51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:36:15.322 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:15.319+0000 7f8098e51640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:36:15.323 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:15.319+0000 7f8098e51640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:36:15.323 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:15.319+0000 7f8098e51640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:36:15.405 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:15.403+0000 7f8098e51640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T06:36:21.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T06:36:21.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:36:21.999 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T06:36:22.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T06:36:22.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:36:22.386 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T06:36:22.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T06:36:22.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:36:22.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:36:22.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:36:22.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:36:22.418 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:22.411+0000 7f7d7d98c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:36:22.420 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:22.415+0000 7f7d7d98c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:36:22.422 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:22.415+0000 7f7d7d98c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:36:22.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T06:36:22.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:36:22.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:36:22.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:36:22.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:36:22.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:22.668 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:36:22.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:36:22.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:22.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:22.976 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:22.971+0000 7f7d7d98c640 -1 Falling back to public interface 2024-02-05T06:36:22.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:23.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:23.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:23.996 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:36:23.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:36:23.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:23.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:24.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:25.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:25.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:25.313 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:36:25.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:36:25.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:25.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:25.487 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:25.483+0000 7f7d7d98c640 -1 osd.2 0 log_to_monitors true 2024-02-05T06:36:25.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:26.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:26.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:26.715 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:36:26.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:36:26.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:26.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:27.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:28.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:28.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:28.030 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:36:28.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:36:28.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:28.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:28.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:29.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:29.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:29.471 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:36:29.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:36:29.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:29.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:29.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:30.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:30.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:30.800 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:36:30.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:36:30.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:30.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:31.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:32.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:32.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:32.159 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:36:32.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:36:32.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:32.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:32.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:33.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:33.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:33.522 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:36:33.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:36:33.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:33.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:33.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:34.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:34.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:34.855 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:36:34.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:36:34.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:34.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:35.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:36.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:36.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:36.333 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:36:36.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:36:36.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:36.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:36.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:37.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:37.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:37.694 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:36:37.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:36:37.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:37.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:38.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:39.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:39.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:39.043 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:36:39.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:36:39.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:39.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:39.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:40.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:40.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:40.740 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:36:40.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:36:40.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:40.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:41.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:42.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:42.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:42.477 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:36:42.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:36:42.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:42.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:42.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:43.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:43.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:43.898 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:36:43.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:36:43.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:43.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:44.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:45.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:45.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:45.626 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:36:45.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:36:45.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:45.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:46.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:47.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:47.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:47.214 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:36:47.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:36:47.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:47.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:47.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:48.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:48.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:48.975 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:36:48.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:36:48.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:48.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:49.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:36:50.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:36:50.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:50.695 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:36:50.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:36:50.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:50.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:36:51.106 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3469783953,v1:127.0.0.1:6819/3469783953] [v2:127.0.0.1:6820/3469783953,v1:127.0.0.1:6821/3469783953] exists,up 553baed5-50fc-49ff-840d-ed74cfa37a16 2024-02-05T06:36:51.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:36:51.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:36:51.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:36:51.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:36:51.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2024-02-05T06:36:51.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:36:51.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:36:51.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T06:36:51.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:36:51.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T06:36:51.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:36:51.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:36:51.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:36:51.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:36:51.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T06:36:51.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T06:36:51.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:36:51.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:36:51.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:36:51.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:36:51.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:36:51.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:36:51.111 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:36:51.112 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:36:51.112 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:36:51.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:36:51.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:36:51.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:36:51.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:36:51.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:36:51.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:36:51.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:36:51.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:36:51.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:36:51.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:36:51.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:36:51.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:36:51.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:36:51.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T06:36:51.113 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:36:51.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9eedd0f9-af5d-4dcc-942a-fc3f057ca686 2024-02-05T06:36:51.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9eedd0f9-af5d-4dcc-942a-fc3f057ca686' 2024-02-05T06:36:51.114 INFO:tasks.workunit.client.0.smithi148.stdout:add osd3 9eedd0f9-af5d-4dcc-942a-fc3f057ca686 2024-02-05T06:36:51.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:36:51.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADgsBlsPb/BxAAP+NHKCOuy+HpvWqvPQF80Q== 2024-02-05T06:36:51.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADgsBlsPb/BxAAP+NHKCOuy+HpvWqvPQF80Q=="}' 2024-02-05T06:36:51.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9eedd0f9-af5d-4dcc-942a-fc3f057ca686 -i td/osd-scrub-repair/3/new.json 2024-02-05T06:36:51.537 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:36:51.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-02-05T06:36:51.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQADgsBlsPb/BxAAP+NHKCOuy+HpvWqvPQF80Q== --osd-uuid 9eedd0f9-af5d-4dcc-942a-fc3f057ca686 2024-02-05T06:36:51.588 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:51.582+0000 7f2492a2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:36:51.621 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:51.614+0000 7f2492a2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:36:51.623 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:51.618+0000 7f2492a2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:36:51.624 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:51.618+0000 7f2492a2a640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T06:36:51.624 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:51.618+0000 7f2492a2a640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T06:36:51.624 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:51.618+0000 7f2492a2a640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T06:36:51.698 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:51.690+0000 7f2492a2a640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-02-05T06:36:58.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-02-05T06:36:58.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:36:58.317 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd3 key to auth repository 2024-02-05T06:36:58.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T06:36:58.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:36:58.738 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.3 2024-02-05T06:36:58.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T06:36:58.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:36:58.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:36:58.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:36:58.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:36:58.770 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:58.762+0000 7f09bc899640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:36:58.772 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:58.766+0000 7f09bc899640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:36:58.773 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:58.766+0000 7f09bc899640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:36:59.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T06:36:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:36:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T06:36:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:36:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:36:59.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:36:59.081 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:36:59.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:36:59.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:36:59.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:36:59.329 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:36:59.322+0000 7f09bc899640 -1 Falling back to public interface 2024-02-05T06:36:59.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:00.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:00.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:00.423 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:37:00.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:37:00.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:00.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:00.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:01.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:01.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:01.721 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:37:01.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:37:01.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:01.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:01.989 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:37:01.982+0000 7f09bc899640 -1 osd.3 0 log_to_monitors true 2024-02-05T06:37:02.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:03.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:03.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:03.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:37:03.032 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:37:03.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:03.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:03.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:04.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:04.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:04.352 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:37:04.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:37:04.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:04.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:04.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:05.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:05.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:05.740 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:37:05.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:37:05.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:05.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:06.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:07.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:07.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:07.043 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:37:07.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:37:07.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:07.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:07.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:08.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:08.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:08.357 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:37:08.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:37:08.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:08.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:08.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:09.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:09.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:09.709 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:37:09.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:37:09.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:09.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:10.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:11.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:11.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:11.117 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:37:11.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:37:11.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:11.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:11.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:12.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:12.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:12.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:37:12.432 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:37:12.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:12.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:13.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:14.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:14.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:14.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:37:14.027 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:37:14.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:14.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:14.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:15.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:15.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:15.348 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:37:15.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:37:15.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:15.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:15.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:16.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:16.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:16.831 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:37:16.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:37:16.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:16.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:17.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:18.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:18.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:18.317 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:37:18.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:37:18.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:18.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:18.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:19.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:19.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:19.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:37:19.650 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:37:19.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:19.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:20.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:21.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:21.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:21.460 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:37:21.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:37:21.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:21.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:21.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:22.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:22.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:22.980 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:37:22.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:37:22.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:22.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:23.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:24.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:24.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:24.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:37:24.628 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:37:24.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:24.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:25.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:26.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:26.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:26.212 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:37:26.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:37:26.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:26.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:26.553 INFO:tasks.workunit.client.0.smithi148.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2130292735,v1:127.0.0.1:6827/2130292735] [v2:127.0.0.1:6828/2130292735,v1:127.0.0.1:6829/2130292735] exists,up 9eedd0f9-af5d-4dcc-942a-fc3f057ca686 2024-02-05T06:37:26.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:37:26.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:37:26.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:37:26.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: create_ec_pool ecpool false k=2 m=2 2024-02-05T06:37:26.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T06:37:26.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T06:37:26.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-02-05T06:37:26.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T06:37:26.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-02-05T06:37:27.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T06:37:27.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T06:37:27.780 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'ecpool' created 2024-02-05T06:37:27.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:37:28.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-02-05T06:37:28.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T06:37:28.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:37:28.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:37:28.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:37:28.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:37:28.803 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:37:28.804 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:37:28.804 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:37:28.804 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:37:28.804 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:37:28.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:37:28.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:37:28.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:37:28.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:37:28.942 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:37:28.942 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:37:29.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:37:29.266 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:37:29.266 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:37:29.266 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T06:37:29.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:37:29.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:29.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:37:29.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-02-05T06:37:29.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-02-05T06:37:29.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-02-05T06:37:29.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:29.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:37:29.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-02-05T06:37:29.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-02-05T06:37:29.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672979' 2024-02-05T06:37:29.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:29.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:37:29.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-02-05T06:37:29.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-02-05T06:37:29.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672979 2-64424509451' 2024-02-05T06:37:29.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:29.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:37:29.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345924 2024-02-05T06:37:29.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345924 2024-02-05T06:37:29.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672979 2-64424509451 3-85899345924' 2024-02-05T06:37:29.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:29.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-02-05T06:37:29.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:29.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:37:29.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-02-05T06:37:29.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:29.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-02-05T06:37:29.913 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836509 2024-02-05T06:37:29.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-02-05T06:37:29.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:37:30.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-02-05T06:37:30.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:30.232 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:30.232 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-02-05T06:37:30.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:37:30.234 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-02-05T06:37:30.234 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:30.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-02-05T06:37:30.236 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672979 2024-02-05T06:37:30.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-02-05T06:37:30.237 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:37:30.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-02-05T06:37:30.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:30.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-02-05T06:37:30.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:30.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:37:30.586 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-02-05T06:37:30.586 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:30.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-02-05T06:37:30.588 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509451 2024-02-05T06:37:30.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-02-05T06:37:30.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:37:30.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509451 2024-02-05T06:37:30.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:37:31.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:37:31.919 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:37:32.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509451 2024-02-05T06:37:32.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:32.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345924 2024-02-05T06:37:32.219 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:32.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:37:32.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345924 2024-02-05T06:37:32.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:32.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345924 2024-02-05T06:37:32.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345924' 2024-02-05T06:37:32.225 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.3 seq 85899345924 2024-02-05T06:37:32.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:37:32.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345925 -lt 85899345924 2024-02-05T06:37:32.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:37:32.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:37:32.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:37:32.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:37:32.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:37:32.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:37:32.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:37:32.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:37:32.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:37:32.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:37:32.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:37:33.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:37:33.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:37:33.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:37:33.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:37:33.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:37:33.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-05T06:37:33.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:37:33.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-05T06:37:33.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:37:33.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:37:33.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:37:33.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:37:33.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:37:33.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:37:33.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:37:33.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:37:33.722 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:37:33.722 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:37:34.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:37:34.012 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:37:34.013 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:37:34.013 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:37:34.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:37:34.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T06:37:34.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:37:34.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:37:34.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:37:34.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:37:34.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:37:34.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:37:34.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:37:34.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:37:34.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:37:34.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:37:34.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:37:34.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:37:34.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:37:34.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:37:34.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:37:34.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:37:34.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:37:34.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:37:34.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:37:34.962 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:37:35.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:37:35.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:37:35.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:37:35.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:37:35.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:37:35.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T06:37:35.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:37:35.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:37:35.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:37:35.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:37:35.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:37:35.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:37:35.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:37:36.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:37:36.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:37:36.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T06:37:36.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:37:36.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T06:37:36.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:37:36.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:37:36.478 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:37:36.478 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:37:36.478 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:37:36.478 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:37:36.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:37:36.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:37:36.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:37:36.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:37:36.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:37:36.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:37:37.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:37:37.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:37:37.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:37:37.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T06:37:37.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-02-05T06:37:37.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:37:37.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:37:37.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T06:37:37.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:37:37.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:37:37.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:37:37.598 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:37:37.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:37:38.142 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:37:38.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:37:38.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:37:38.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T06:37:38.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: get_primary ecpool SOMETHING 2024-02-05T06:37:38.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T06:37:38.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T06:37:38.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T06:37:38.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:37:38.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: local primary=1 2024-02-05T06:37:38.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: get_osds ecpool SOMETHING 2024-02-05T06:37:38.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: sed -e s/1// 2024-02-05T06:37:38.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T06:37:38.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-02-05T06:37:38.833 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-02-05T06:37:38.834 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:37:39.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T06:37:39.140 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:37:39.140 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:37:39.140 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T06:37:39.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 3 2024-02-05T06:37:39.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: osds=('0' '2' '3') 2024-02-05T06:37:39.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: local -a osds 2024-02-05T06:37:39.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: local not_primary_first=0 2024-02-05T06:37:39.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: local not_primary_second=2 2024-02-05T06:37:39.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:871: unfound_erasure_coded: local not_primary_third=3 2024-02-05T06:37:39.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: pids= 2024-02-05T06:37:39.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:37:39.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T06:37:39.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T06:37:39.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 462138"' 2024-02-05T06:37:39.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 462138' 2024-02-05T06:37:39.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:37:39.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T06:37:39.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T06:37:39.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 462139"' 2024-02-05T06:37:39.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 462139' 2024-02-05T06:37:39.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:37:39.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T06:37:39.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T06:37:39.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:37:39.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 462143"' 2024-02-05T06:37:39.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/462141: /' 2024-02-05T06:37:39.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:37:39.145 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 462143' 2024-02-05T06:37:39.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: wait_background pids 2024-02-05T06:37:39.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 462138 462139 462143' 2024-02-05T06:37:39.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T06:37:39.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T06:37:39.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 462138 2024-02-05T06:37:39.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/462144: /' 2024-02-05T06:37:39.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:37:39.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/462147: /' 2024-02-05T06:37:42.281 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:37:42.282 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:37:42.282 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T06:37:42.282 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:37:42.282 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:37:42.282 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:37:42.282 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:37:42.282 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T06:37:42.282 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:37:42.282 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T06:37:42.282 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:37:42.283 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:37:42.283 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:37:42.283 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:37:42.283 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:37:42.283 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:37:42.283 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:37:42.283 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:37:42.283 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:37:42.283 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T06:37:42.283 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:37:42.283 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:37:42.283 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-02-05T06:37:42.284 INFO:tasks.workunit.client.0.smithi148.stderr:462144: remove 2#1:eb822e21:::SOMETHING:head# 2024-02-05T06:37:42.284 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T06:37:42.284 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:37:42.284 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:37:42.284 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T06:37:42.284 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:37:42.284 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:37:42.284 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:37:42.284 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:37:42.288 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:37:42.288 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:37:42.288 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:37:42.288 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:37:42.288 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:37:42.288 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:37:42.288 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:37:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:37:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:37:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:37:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:462144: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:37:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:462144: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:37:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:462144: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:37:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:37:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:37:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:37:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:37:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:37:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:37:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:37:42.289 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:37:42.290 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:37:42.290 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:37:42.290 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:37:42.290 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:37:42.290 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T06:37:42.290 INFO:tasks.workunit.client.0.smithi148.stderr:462144: start osd.2 2024-02-05T06:37:42.290 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:37:42.346 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:37:42.347 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:37:42.347 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T06:37:42.347 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:37:42.347 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:37:42.347 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:37:42.347 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:37:42.347 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T06:37:42.347 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:37:42.347 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T06:37:42.347 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:37:42.347 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:37:42.348 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:37:42.348 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:37:42.348 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:37:42.348 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:37:42.348 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:37:42.348 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:37:42.348 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:37:42.348 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T06:37:42.348 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:37:42.348 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T06:37:42.348 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T06:37:42.349 INFO:tasks.workunit.client.0.smithi148.stderr:462147: remove 3#1:eb822e21:::SOMETHING:head# 2024-02-05T06:37:42.349 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T06:37:42.349 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:37:42.349 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:37:42.349 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T06:37:42.349 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:37:42.349 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T06:37:42.349 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:37:42.349 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:37:42.350 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:37:42.350 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:37:42.350 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T06:37:42.350 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T06:37:42.351 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:37:42.351 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:37:42.351 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:37:42.351 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:37:42.351 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:37:42.351 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:37:42.351 INFO:tasks.workunit.client.0.smithi148.stderr:462147: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:37:42.351 INFO:tasks.workunit.client.0.smithi148.stderr:462147: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:37:42.351 INFO:tasks.workunit.client.0.smithi148.stderr:462147: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:37:42.351 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:37:42.351 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:37:42.351 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:37:42.351 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:37:42.351 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:37:42.352 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:37:42.352 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:37:42.352 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:37:42.352 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:37:42.352 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:37:42.352 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:37:42.352 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T06:37:42.352 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T06:37:42.352 INFO:tasks.workunit.client.0.smithi148.stderr:462147: start osd.3 2024-02-05T06:37:42.352 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:37:42.407 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:37:42.407 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:37:42.407 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:37:42.407 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:37:42.407 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:37:42.407 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:37:42.408 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:37:42.408 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:37:42.408 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:37:42.408 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:37:42.408 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:37:42.408 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:37:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:37:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:37:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:37:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:37:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:37:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:37:42.409 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:37:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:37:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:37:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:37:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-02-05T06:37:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:462141: remove 1#1:eb822e21:::SOMETHING:head# 2024-02-05T06:37:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:37:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:37:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:37:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:37:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:37:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:37:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:37:42.410 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:37:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:37:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:37:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:37:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:37:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:37:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:37:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:37:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:37:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:37:42.412 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:37:42.412 INFO:tasks.workunit.client.0.smithi148.stderr:462141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:37:42.412 INFO:tasks.workunit.client.0.smithi148.stderr:462141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:37:42.412 INFO:tasks.workunit.client.0.smithi148.stderr:462141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:37:42.412 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:37:42.412 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:37:42.412 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:37:42.412 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:37:42.412 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:37:42.412 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:37:42.412 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:37:42.412 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:37:42.412 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:37:42.413 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:37:42.413 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:37:42.413 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:37:42.413 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:37:42.413 INFO:tasks.workunit.client.0.smithi148.stderr:462141: start osd.0 2024-02-05T06:37:42.413 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:37:47.009 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T06:37:47.009 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T06:37:47.010 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:37:47.010 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:37:47.010 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:37:47.010 INFO:tasks.workunit.client.0.smithi148.stderr:462144: 2024-02-05T06:37:42.317+0000 7f044a4f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:37:47.010 INFO:tasks.workunit.client.0.smithi148.stderr:462144: 2024-02-05T06:37:42.321+0000 7f044a4f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:37:47.010 INFO:tasks.workunit.client.0.smithi148.stderr:462144: 2024-02-05T06:37:42.321+0000 7f044a4f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:37:47.010 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T06:37:47.010 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:37:47.010 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:37:47.010 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:37:47.010 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:37:47.010 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:47.010 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:37:47.011 INFO:tasks.workunit.client.0.smithi148.stderr:462144: 0 2024-02-05T06:37:47.011 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:47.011 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:37:47.011 INFO:tasks.workunit.client.0.smithi148.stderr:462144: 2024-02-05T06:37:42.873+0000 7f044a4f9640 -1 Falling back to public interface 2024-02-05T06:37:47.011 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:47.011 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:47.011 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:47.011 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:37:47.011 INFO:tasks.workunit.client.0.smithi148.stderr:462144: 1 2024-02-05T06:37:47.011 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:47.011 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:37:47.011 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:37:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:462144: 2 2024-02-05T06:37:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:37:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:462144: 2024-02-05T06:37:46.065+0000 7f044a4f9640 -1 osd.2 28 log_to_monitors true 2024-02-05T06:37:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:37:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:462144: 3 2024-02-05T06:37:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:37:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:47.013 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T06:37:47.013 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T06:37:47.013 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:37:47.013 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:37:47.013 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:37:47.013 INFO:tasks.workunit.client.0.smithi148.stderr:462147: 2024-02-05T06:37:42.373+0000 7f83188dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:37:47.013 INFO:tasks.workunit.client.0.smithi148.stderr:462147: 2024-02-05T06:37:42.377+0000 7f83188dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:37:47.013 INFO:tasks.workunit.client.0.smithi148.stderr:462147: 2024-02-05T06:37:42.377+0000 7f83188dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:37:47.013 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T06:37:47.013 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:37:47.013 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T06:37:47.013 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:37:47.014 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:37:47.014 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:47.014 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:37:47.014 INFO:tasks.workunit.client.0.smithi148.stderr:462147: 0 2024-02-05T06:37:47.014 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:47.014 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:47.014 INFO:tasks.workunit.client.0.smithi148.stderr:462147: 2024-02-05T06:37:42.941+0000 7f83188dc640 -1 Falling back to public interface 2024-02-05T06:37:47.014 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:47.014 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:47.014 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:47.014 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:37:47.014 INFO:tasks.workunit.client.0.smithi148.stderr:462147: 1 2024-02-05T06:37:47.014 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:47.014 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:47.015 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:47.015 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:47.015 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:47.015 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:37:47.015 INFO:tasks.workunit.client.0.smithi148.stderr:462147: 2 2024-02-05T06:37:47.015 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:47.015 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:47.015 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:47.015 INFO:tasks.workunit.client.0.smithi148.stderr:462147: 2024-02-05T06:37:46.301+0000 7f83188dc640 -1 osd.3 28 log_to_monitors true 2024-02-05T06:37:47.015 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:47.015 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:47.015 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:37:47.015 INFO:tasks.workunit.client.0.smithi148.stderr:462147: 3 2024-02-05T06:37:47.015 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:47.016 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:47.241 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:37:47.241 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:37:47.241 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:37:47.241 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:37:47.241 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:37:47.241 INFO:tasks.workunit.client.0.smithi148.stderr:462141: 2024-02-05T06:37:42.437+0000 7f5712c7d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:37:47.241 INFO:tasks.workunit.client.0.smithi148.stderr:462141: 2024-02-05T06:37:42.441+0000 7f5712c7d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:37:47.242 INFO:tasks.workunit.client.0.smithi148.stderr:462141: 2024-02-05T06:37:42.445+0000 7f5712c7d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:37:47.242 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:37:47.242 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:37:47.242 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:37:47.242 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:37:47.242 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:37:47.242 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:47.242 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:37:47.242 INFO:tasks.workunit.client.0.smithi148.stderr:462141: 0 2024-02-05T06:37:47.242 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:47.242 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:37:47.242 INFO:tasks.workunit.client.0.smithi148.stderr:462141: 2024-02-05T06:37:43.009+0000 7f5712c7d640 -1 Falling back to public interface 2024-02-05T06:37:47.242 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:47.243 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:47.243 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:47.243 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:37:47.243 INFO:tasks.workunit.client.0.smithi148.stderr:462141: 1 2024-02-05T06:37:47.243 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:47.243 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:37:47.243 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:47.243 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:47.243 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:47.243 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:37:47.243 INFO:tasks.workunit.client.0.smithi148.stderr:462141: 2 2024-02-05T06:37:47.243 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:47.243 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:37:47.244 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:47.244 INFO:tasks.workunit.client.0.smithi148.stderr:462141: 2024-02-05T06:37:46.441+0000 7f5712c7d640 -1 osd.0 28 log_to_monitors true 2024-02-05T06:37:47.244 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:47.244 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:47.244 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:37:47.244 INFO:tasks.workunit.client.0.smithi148.stderr:462141: 3 2024-02-05T06:37:47.244 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:47.244 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:37:48.883 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:48.884 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:48.884 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:48.884 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:37:48.884 INFO:tasks.workunit.client.0.smithi148.stderr:462147: 4 2024-02-05T06:37:48.884 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:48.884 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:37:48.884 INFO:tasks.workunit.client.0.smithi148.stderr:462147: osd.3 up in weight 1 up_from 34 up_thru 0 down_at 29 last_clean_interval [20,28) [v2:127.0.0.1:6818/2438934165,v1:127.0.0.1:6819/2438934165] [v2:127.0.0.1:6820/2438934165,v1:127.0.0.1:6821/2438934165] exists,up 9eedd0f9-af5d-4dcc-942a-fc3f057ca686 2024-02-05T06:37:48.884 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:37:48.884 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:37:48.884 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:37:48.884 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:37:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:37:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:37:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:37:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:37:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:462147: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:37:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:462147: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:37:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:37:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:37:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:37:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:37:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:37:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:37:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:37:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:37:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:37:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:37:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:462147: 1 2024-02-05T06:37:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:462147: 2 2024-02-05T06:37:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:462147: 3' 2024-02-05T06:37:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:37:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:37:48.887 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-02-05T06:37:48.887 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-02-05T06:37:48.887 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066' 2024-02-05T06:37:48.887 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:48.887 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hs.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:48.887 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:48.887 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:48.887 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:37:48.888 INFO:tasks.workunit.client.0.smithi148.stderr:462144: 4 2024-02-05T06:37:48.888 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:48.888 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:37:48.888 INFO:tasks.workunit.client.0.smithi148.stderr:462144: osd.2 up in weight 1 up_from 33 up_thru 0 down_at 29 last_clean_interval [15,28) [v2:127.0.0.1:6802/1875205,v1:127.0.0.1:6803/1875205] [v2:127.0.0.1:6804/1875205,v1:127.0.0.1:6805/1875205] exists,up 553baed5-50fc-49ff-840d-ed74cfa37a16 2024-02-05T06:37:48.888 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:37:48.888 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:37:48.888 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:37:48.888 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:37:48.888 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:37:48.888 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:37:48.888 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:37:48.888 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:37:48.889 INFO:tasks.workunit.client.0.smithi148.stderr:462144: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:37:48.889 INFO:tasks.workunit.client.0.smithi148.stderr:462144: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:37:48.889 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:37:48.889 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:37:48.889 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:37:48.889 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:37:48.889 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:37:48.889 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:37:48.889 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:37:48.889 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:37:48.890 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:37:48.890 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:37:48.890 INFO:tasks.workunit.client.0.smithi148.stderr:462144: 1 2024-02-05T06:37:48.890 INFO:tasks.workunit.client.0.smithi148.stderr:462144: 2 2024-02-05T06:37:48.890 INFO:tasks.workunit.client.0.smithi148.stderr:462144: 3' 2024-02-05T06:37:48.890 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:37:48.890 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:48.891 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:37:48.891 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888067 2024-02-05T06:37:48.891 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888067 2024-02-05T06:37:48.891 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888067' 2024-02-05T06:37:48.891 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:49.154 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22s.sh:985: wait_for_osd: sleep 1 2024-02-05T06:37:49.154 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:37:49.154 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:37:49.155 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:37:49.155 INFO:tasks.workunit.client.0.smithi148.stderr:462141: 4 2024-02-05T06:37:49.155 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:37:49.155 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:37:49.155 INFO:tasks.workunit.client.0.smithi148.stderr:462141: osd.0 up in weight 1 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6826/485365968,v1:127.0.0.1:6827/485365968] [v2:127.0.0.1:6828/485365968,v1:127.0.0.1:6829/485365968] exists,up bb578e5b-a58d-4d6e-b434-b7a6746025fe 2024-02-05T06:37:49.155 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:37:49.155 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:37:49.155 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:37:49.155 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:37:49.155 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:37:49.155 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:37:49.155 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:37:49.156 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:37:49.156 INFO:tasks.workunit.client.0.smithi148.stderr:462141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:37:49.156 INFO:tasks.workunit.client.0.smithi148.stderr:462141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:37:49.156 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:37:49.156 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:37:49.156 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:37:49.156 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:37:49.156 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:37:49.156 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:37:49.156 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:37:49.156 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:37:49.157 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:37:49.157 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:37:49.157 INFO:tasks.workunit.client.0.smithi148.stderr:462141: 1 2024-02-05T06:37:49.157 INFO:tasks.workunit.client.0.smithi148.stderr:462141: 2 2024-02-05T06:37:49.157 INFO:tasks.workunit.client.0.smithi148.stderr:462141: 3' 2024-02-05T06:37:49.157 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:37:49.157 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:49.157 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:37:49.157 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888068 2024-02-05T06:37:49.157 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888068 2024-02-05T06:37:49.157 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888068' 2024-02-05T06:37:49.158 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:50.169 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:37:50.169 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-02-05T06:37:50.170 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-02-05T06:37:50.170 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888068 1-42949672986' 2024-02-05T06:37:50.170 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:50.170 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:37:50.170 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920772 2024-02-05T06:37:50.170 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920772 2024-02-05T06:37:50.170 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888068 1-42949672986 2-141733920772' 2024-02-05T06:37:50.170 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:50.170 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:37:50.170 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888068 2024-02-05T06:37:50.171 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888068 2024-02-05T06:37:50.171 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888068 1-42949672986 2-141733920772 3-146028888068' 2024-02-05T06:37:50.171 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:50.171 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888068 2024-02-05T06:37:50.171 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:50.171 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:37:50.171 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888068 2024-02-05T06:37:50.171 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:50.171 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888068 2024-02-05T06:37:50.172 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888068' 2024-02-05T06:37:50.172 INFO:tasks.workunit.client.0.smithi148.stderr:462141: waiting osd.0 seq 146028888068 2024-02-05T06:37:50.172 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:37:50.172 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888068 2024-02-05T06:37:50.172 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:50.172 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-02-05T06:37:50.172 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:50.172 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:37:50.172 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-02-05T06:37:50.173 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:50.173 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-02-05T06:37:50.173 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-02-05T06:37:50.173 INFO:tasks.workunit.client.0.smithi148.stderr:462141: waiting osd.1 seq 42949672986 2024-02-05T06:37:50.173 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:37:50.919 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: elpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:37:50.919 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-02-05T06:37:50.919 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-02-05T06:37:50.919 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-42949672984' 2024-02-05T06:37:50.919 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:50.919 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:37:50.919 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920770 2024-02-05T06:37:50.919 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920770 2024-02-05T06:37:50.919 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-42949672984 2-141733920770' 2024-02-05T06:37:50.919 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:50.919 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:37:50.920 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-02-05T06:37:50.920 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-02-05T06:37:50.920 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-42949672984 2-141733920770 3-146028888066' 2024-02-05T06:37:50.920 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:50.920 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888066 2024-02-05T06:37:50.920 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:50.920 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:37:50.920 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888066 2024-02-05T06:37:50.920 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:50.920 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-02-05T06:37:50.920 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-02-05T06:37:50.920 INFO:tasks.workunit.client.0.smithi148.stderr:462147: waiting osd.0 seq 146028888066 2024-02-05T06:37:50.920 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:37:50.920 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 146028888066 2024-02-05T06:37:50.921 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:37:50.921 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:37:50.921 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:37:50.921 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888066 2024-02-05T06:37:50.921 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:50.921 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-02-05T06:37:50.921 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:50.921 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:37:50.921 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-02-05T06:37:50.921 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:50.960 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubunt02: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:37:50.960 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-02-05T06:37:50.960 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-02-05T06:37:50.961 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888067 1-42949672985' 2024-02-05T06:37:50.961 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:50.961 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:37:50.961 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920771 2024-02-05T06:37:50.961 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920771 2024-02-05T06:37:50.961 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888067 1-42949672985 2-141733920771' 2024-02-05T06:37:50.961 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:37:50.961 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:37:50.961 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888067 2024-02-05T06:37:50.961 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888067 2024-02-05T06:37:50.961 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888067 1-42949672985 2-141733920771 3-146028888067' 2024-02-05T06:37:50.961 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:50.961 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888067 2024-02-05T06:37:50.961 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:50.962 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:37:50.962 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888067 2024-02-05T06:37:50.962 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:50.962 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888067 2024-02-05T06:37:50.962 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2024-02-05T06:37:50.962 INFO:tasks.workunit.client.0.smithi148.stderr:462144: waiting osd.0 seq 146028888067 2024-02-05T06:37:50.962 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:37:50.962 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 146028888067 2024-02-05T06:37:50.962 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:37:50.962 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:37:50.962 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:37:50.962 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888067 2024-02-05T06:37:50.962 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:50.962 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-02-05T06:37:50.963 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:50.963 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:37:50.963 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-02-05T06:37:50.963 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:51.584 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/cflush_pg_stats: test 42949672986 -lt 42949672986 2024-02-05T06:37:51.584 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:51.584 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920772 2024-02-05T06:37:51.585 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:51.585 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:37:51.585 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920772 2024-02-05T06:37:51.585 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:51.585 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920772 2024-02-05T06:37:51.585 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920772' 2024-02-05T06:37:51.585 INFO:tasks.workunit.client.0.smithi148.stderr:462141: waiting osd.2 seq 141733920772 2024-02-05T06:37:51.585 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:37:51.585 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920772 2024-02-05T06:37:51.585 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:51.585 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:51.585 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-146028888068 2024-02-05T06:37:51.585 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:37:51.586 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-146028888068 2024-02-05T06:37:51.586 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:51.586 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888068 2024-02-05T06:37:51.586 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 146028888068' 2024-02-05T06:37:51.586 INFO:tasks.workunit.client.0.smithi148.stderr:462141: waiting osd.3 seq 146028888068 2024-02-05T06:37:51.586 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:37:51.586 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888068 2024-02-05T06:37:51.586 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:37:51.586 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:37:51.586 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:37:51.586 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:37:51.586 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:37:51.586 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:37:51.587 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:37:51.587 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:37:51.587 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:37:51.587 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:37:51.587 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:37:51.966 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:37:51.967 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:37:51.967 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:37:51.967 INFO:tasks.workunit.client.0.smithi148.stderr:462141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:37:51.967 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:37:51.967 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:37:51.967 INFO:tasks.workunit.client.0.smithi148.stderr:462141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:37:51.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T06:37:51.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T06:37:51.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 462139 2024-02-05T06:37:52.272 INFO:tasks.workunit.client.0.smithi148.stderr:u/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-02-05T06:37:52.272 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-02-05T06:37:52.272 INFO:tasks.workunit.client.0.smithi148.stderr:462147: waiting osd.1 seq 42949672984 2024-02-05T06:37:52.272 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:37:52.272 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672984 2024-02-05T06:37:52.272 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:52.272 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920770 2024-02-05T06:37:52.272 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:52.272 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:37:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920770 2024-02-05T06:37:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920770 2024-02-05T06:37:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920770' 2024-02-05T06:37:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:462147: waiting osd.2 seq 141733920770 2024-02-05T06:37:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:37:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920770 2024-02-05T06:37:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-146028888066 2024-02-05T06:37:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:37:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-146028888066 2024-02-05T06:37:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:52.274 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-02-05T06:37:52.274 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 146028888066' 2024-02-05T06:37:52.274 INFO:tasks.workunit.client.0.smithi148.stderr:462147: waiting osd.3 seq 146028888066 2024-02-05T06:37:52.274 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:37:52.274 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888066 2024-02-05T06:37:52.274 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:37:52.274 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:37:52.274 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:37:52.274 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:37:52.274 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:37:52.274 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:37:52.274 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:37:52.274 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:37:52.429 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") |lone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-02-05T06:37:52.429 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-02-05T06:37:52.429 INFO:tasks.workunit.client.0.smithi148.stderr:462144: waiting osd.1 seq 42949672985 2024-02-05T06:37:52.429 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:37:52.429 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672985 2024-02-05T06:37:52.429 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:52.430 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920771 2024-02-05T06:37:52.430 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:52.430 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:37:52.430 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920771 2024-02-05T06:37:52.430 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:52.430 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920771 2024-02-05T06:37:52.430 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2024-02-05T06:37:52.430 INFO:tasks.workunit.client.0.smithi148.stderr:462144: waiting osd.2 seq 141733920771 2024-02-05T06:37:52.430 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:37:52.430 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920771 2024-02-05T06:37:52.430 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:37:52.430 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-146028888067 2024-02-05T06:37:52.430 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:37:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:37:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-146028888067 2024-02-05T06:37:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:37:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888067 2024-02-05T06:37:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 146028888067' 2024-02-05T06:37:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:462144: waiting osd.3 seq 146028888067 2024-02-05T06:37:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:37:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888067 2024-02-05T06:37:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:37:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:37:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:37:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:37:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:37:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:37:52.432 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:37:52.432 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:37:52.432 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:37:52.946 INFO:tasks.workunit.client.0.smithi148.stderr:46214 not)' 2024-02-05T06:37:52.947 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:37:52.947 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:37:52.947 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:37:52.947 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:37:52.947 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:37:52.947 INFO:tasks.workunit.client.0.smithi148.stderr:462147: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:37:52.947 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:37:52.947 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:37:52.947 INFO:tasks.workunit.client.0.smithi148.stderr:462147: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:37:52.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T06:37:53.116 INFO:tasks.workunit.client.0.smithi148.stderr:4: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:37:53.117 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:37:53.117 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:37:53.117 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:37:53.117 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:37:53.117 INFO:tasks.workunit.client.0.smithi148.stderr:462144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:37:53.117 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:37:53.117 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:37:53.117 INFO:tasks.workunit.client.0.smithi148.stderr:462144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:37:53.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T06:37:53.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T06:37:53.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 462143 2024-02-05T06:37:53.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T06:37:53.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T06:37:53.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T06:37:53.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: return_code=0 2024-02-05T06:37:53.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:882: unfound_erasure_coded: '[' 0 -ne 0 ']' 2024-02-05T06:37:53.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: get_pg ecpool SOMETHING 2024-02-05T06:37:53.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T06:37:53.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T06:37:53.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T06:37:53.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:37:53.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: local pg=1.0 2024-02-05T06:37:53.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: unfound_erasure_coded: repair 1.0 2024-02-05T06:37:53.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-02-05T06:37:53.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-02-05T06:37:53.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:37:53.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:37:53.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:37:53.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:37:53.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T06:37:27.635733+0000 2024-02-05T06:37:53.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-02-05T06:37:54.047 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-02-05T06:37:54.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-02-05T06:37:27.635733+0000 2024-02-05T06:37:54.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:37:54.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:37:27.635733+0000 2024-02-05T06:37:54.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:37:54.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:37:54.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:37:54.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:37:54.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:37:54.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:37:54.065 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:37:54.065 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:37:54.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:37:27.635733+0000 '>' 2024-02-05T06:37:27.635733+0000 2024-02-05T06:37:54.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:37:55.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:37:55.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:37:55.373 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:37:55.373 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:37:55.373 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:37:55.374 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:37:55.374 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:37:55.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:37:27.635733+0000 '>' 2024-02-05T06:37:27.635733+0000 2024-02-05T06:37:55.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:37:56.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:37:56.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:37:56.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:37:56.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:37:56.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:37:56.687 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:37:56.687 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:37:56.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:37:27.635733+0000 '>' 2024-02-05T06:37:27.635733+0000 2024-02-05T06:37:56.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:37:57.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:37:57.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:37:57.994 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:37:57.994 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:37:57.994 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:37:57.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:37:57.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:37:58.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:37:27.635733+0000 '>' 2024-02-05T06:37:27.635733+0000 2024-02-05T06:37:58.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:37:59.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:37:59.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:37:59.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:37:59.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:37:59.300 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:37:59.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:37:59.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:37:59.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:37:27.635733+0000 '>' 2024-02-05T06:37:27.635733+0000 2024-02-05T06:37:59.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:38:00.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:38:00.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:38:00.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:38:00.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:38:00.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:38:00.616 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:38:00.616 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:38:00.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:37:54.607869+0000 '>' 2024-02-05T06:37:27.635733+0000 2024-02-05T06:38:00.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:38:00.931 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: seq 1 60 2024-02-05T06:38:00.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: for f in `seq 1 60` 2024-02-05T06:38:00.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: ceph -s 2024-02-05T06:38:00.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: grep '1/1 objects unfound' 2024-02-05T06:38:01.278 INFO:tasks.workunit.client.0.smithi148.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:38:01.278 INFO:tasks.workunit.client.0.smithi148.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:38:01.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: break 2024-02-05T06:38:01.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: ceph -s 2024-02-05T06:38:01.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: grep '4 up' 2024-02-05T06:38:01.653 INFO:tasks.workunit.client.0.smithi148.stdout: osd: 4 osds: 4 up (since 14s), 4 in (since 70s) 2024-02-05T06:38:01.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: grep '4 in' 2024-02-05T06:38:01.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: ceph -s 2024-02-05T06:38:02.152 INFO:tasks.workunit.client.0.smithi148.stdout: osd: 4 osds: 4 up (since 14s), 4 in (since 70s) 2024-02-05T06:38:02.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: ceph -s 2024-02-05T06:38:02.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: grep '1/1 objects unfound' 2024-02-05T06:38:02.530 INFO:tasks.workunit.client.0.smithi148.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:38:02.530 INFO:tasks.workunit.client.0.smithi148.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:38:02.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:38:02.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:38:02.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:38:02.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:38:02.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:38:02.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:38:02.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:38:02.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:38:02.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:38:02.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:38:02.845 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:38:02.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:38:02.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:38:02.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:38:02.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:38:02.849 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:38:02.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:38:02.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:38:02.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:38:02.852 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:38:02.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:38:02.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:38:02.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:38:02.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:38:02.883 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:38:02.883 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:38:02.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:38:02.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:38:02.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:38:02.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:38:02.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:38:02.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:38:02.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:38:02.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:38:02.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:38:02.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:38:02.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:38:02.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:38:02.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:38:02.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:38:02.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:38:02.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:38:02.889 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:38:02.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:38:02.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:38:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:38:02.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:38:02.893 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:38:02.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:38:02.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:38:02.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:38:02.894 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:38:02.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:38:02.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:38:02.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:38:02.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:38:02.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:38:02.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:38:02.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:38:02.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:38:02.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:38:02.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:38:02.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:38:02.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:38:02.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:38:02.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61355 2024-02-05T06:38:02.906 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:38:02.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:38:02.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:38:02.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:38:02.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_overwrites td/osd-scrub-repair 2024-02-05T06:38:02.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: TEST_unfound_erasure_coded_overwrites: '[' true = true ']' 2024-02-05T06:38:02.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:908: TEST_unfound_erasure_coded_overwrites: unfound_erasure_coded td/osd-scrub-repair true 2024-02-05T06:38:02.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: unfound_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T06:38:02.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: unfound_erasure_coded: local allow_overwrites=true 2024-02-05T06:38:02.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:854: unfound_erasure_coded: local poolname=ecpool 2024-02-05T06:38:02.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:855: unfound_erasure_coded: local payload=ABCDEF 2024-02-05T06:38:02.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2024-02-05T06:38:02.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:38:02.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:38:02.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:38:02.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:38:02.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:38:02.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T06:38:03.307 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:38:03.308 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:38:03.308 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:38:03.308 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:38:03.308 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:38:03.308 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:38:03.309 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:38:03.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T06:38:03.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:38:03.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:38:03.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:38:03.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:38:03.510 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:38:03.510 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:38:03.510 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:38:03.511 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:38:03.511 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:38:03.511 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:38:03.511 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:38:03.511 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:38:03.512 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:38:03.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:38:03.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get fsid 2024-02-05T06:38:03.629 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:38:03.629 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:38:03.629 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:38:03.629 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:38:03.630 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:38:03.630 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:38:03.630 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:38:03.631 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:38:03.631 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:38:03.631 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:38:03.631 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:38:03.632 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61355/ceph-mon.a.asok 2024-02-05T06:38:03.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:38:03.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61355/ceph-mon.a.asok config get mon_host 2024-02-05T06:38:03.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2024-02-05T06:38:03.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:38:03.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:38:03.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:38:03.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:38:03.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:38:03.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:38:04.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:38:04.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:38:04.609 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:38:04.610 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:38:04.610 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:38:04.610 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:38:04.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:38:04.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:38:04.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:38:04.646 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: seq 0 3 2024-02-05T06:38:04.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:38:04.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2024-02-05T06:38:04.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:38:04.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:38:04.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:38:04.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:38:04.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:38:04.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:38:04.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:38:04.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:38:04.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:38:04.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:38:04.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:38:04.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:38:04.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:38:04.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:38:04.649 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:38:04.649 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:38:04.649 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:38:04.649 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:38:04.650 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:38:04.650 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:38:04.650 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:38:04.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:38:04.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:38:04.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:38:04.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:38:04.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:38:04.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:38:04.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:38:04.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:38:04.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:38:04.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:38:04.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:38:04.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:38:04.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:38:04.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:38:04.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7e7dc194-3d95-40f8-8258-f3a08cc6a777 2024-02-05T06:38:04.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7e7dc194-3d95-40f8-8258-f3a08cc6a777' 2024-02-05T06:38:04.653 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 7e7dc194-3d95-40f8-8258-f3a08cc6a777 2024-02-05T06:38:04.653 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:38:04.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBMgsBlE4fIJxAA+sRlp2DF7cLgTtKvfVhd/w== 2024-02-05T06:38:04.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBMgsBlE4fIJxAA+sRlp2DF7cLgTtKvfVhd/w=="}' 2024-02-05T06:38:04.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7e7dc194-3d95-40f8-8258-f3a08cc6a777 -i td/osd-scrub-repair/0/new.json 2024-02-05T06:38:04.930 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:38:04.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:38:04.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMgsBlE4fIJxAA+sRlp2DF7cLgTtKvfVhd/w== --osd-uuid 7e7dc194-3d95-40f8-8258-f3a08cc6a777 2024-02-05T06:38:04.983 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:04.977+0000 7f132122b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:38:04.985 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:04.981+0000 7f132122b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:38:04.986 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:04.981+0000 7f132122b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:38:04.987 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:04.981+0000 7f132122b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:38:04.987 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:04.981+0000 7f132122b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:38:04.987 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:04.981+0000 7f132122b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:38:05.073 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:05.069+0000 7f132122b640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:38:11.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:38:11.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:38:11.022 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:38:11.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:38:11.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:38:11.386 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:38:11.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:38:11.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:38:11.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:38:11.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:38:11.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:38:11.418 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:11.412+0000 7f7cdcb5c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:38:11.420 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:11.416+0000 7f7cdcb5c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:38:11.421 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:11.416+0000 7f7cdcb5c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:38:11.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:38:11.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:38:11.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:38:11.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:38:11.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:38:11.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:11.716 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:38:11.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:38:11.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:11.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:11.986 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:11.980+0000 7f7cdcb5c640 -1 Falling back to public interface 2024-02-05T06:38:12.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:13.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:13.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:13.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:38:13.030 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:38:13.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:13.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:13.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:14.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:14.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:14.331 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:38:14.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:38:14.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:14.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:14.582 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:14.576+0000 7f7cdcb5c640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:38:14.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:15.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:15.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:15.651 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:38:15.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:38:15.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:15.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:15.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:16.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:16.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:16.969 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:38:16.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:38:16.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:16.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:17.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:18.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:18.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:18.437 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:38:18.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:38:18.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:18.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:18.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:19.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:19.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:19.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:38:19.754 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:38:19.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:19.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:20.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:21.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:21.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:21.079 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:38:21.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:38:21.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:21.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:21.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:22.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:22.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:22.588 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:38:22.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:38:22.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:22.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:22.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:23.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:23.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:23.900 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:38:23.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:38:23.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:23.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:24.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:25.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:25.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:25.349 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:38:25.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:38:25.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:25.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:25.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:26.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:26.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:26.668 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:38:26.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:38:26.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:26.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:26.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:27.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:27.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:27.993 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:38:27.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:38:27.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:27.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:28.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:29.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:29.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:29.647 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:38:29.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:38:29.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:29.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:30.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:31.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:31.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:31.264 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:38:31.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:38:31.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:31.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:31.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:32.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:32.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:32.879 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:38:32.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:38:32.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:32.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:33.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:34.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:34.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:34.481 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:38:34.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:38:34.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:34.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:35.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:36.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:36.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:36.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:38:36.152 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:38:36.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:36.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:36.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:37.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:37.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:37.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:38:37.682 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:38:37.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:37.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:38.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:39.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:39.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:39.161 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:38:39.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:38:39.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:39.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:38:39.478 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/999179860,v1:127.0.0.1:6803/999179860] [v2:127.0.0.1:6804/999179860,v1:127.0.0.1:6805/999179860] exists,up 7e7dc194-3d95-40f8-8258-f3a08cc6a777 2024-02-05T06:38:39.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:38:39.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:38:39.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:38:39.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:38:39.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2024-02-05T06:38:39.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:38:39.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:38:39.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:38:39.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:38:39.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:38:39.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:38:39.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:38:39.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:38:39.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:38:39.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:38:39.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:38:39.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:38:39.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:38:39.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:38:39.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:38:39.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:38:39.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:38:39.483 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:38:39.483 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:38:39.483 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:38:39.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:38:39.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:38:39.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:38:39.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:38:39.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:38:39.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:38:39.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:38:39.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:38:39.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:38:39.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:38:39.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:38:39.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:38:39.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:38:39.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:38:39.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:38:39.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=74a0d2be-74f2-41da-b18b-b8b00642c2b3 2024-02-05T06:38:39.485 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 74a0d2be-74f2-41da-b18b-b8b00642c2b3 2024-02-05T06:38:39.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 74a0d2be-74f2-41da-b18b-b8b00642c2b3' 2024-02-05T06:38:39.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:38:39.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvgsBl1hoLHhAAeb5QIeHf+P3KBPC5pEcMgg== 2024-02-05T06:38:39.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvgsBl1hoLHhAAeb5QIeHf+P3KBPC5pEcMgg=="}' 2024-02-05T06:38:39.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 74a0d2be-74f2-41da-b18b-b8b00642c2b3 -i td/osd-scrub-repair/1/new.json 2024-02-05T06:38:39.868 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:38:39.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T06:38:39.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvgsBl1hoLHhAAeb5QIeHf+P3KBPC5pEcMgg== --osd-uuid 74a0d2be-74f2-41da-b18b-b8b00642c2b3 2024-02-05T06:38:39.926 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:39.920+0000 7f9c4ff42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:38:39.953 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:39.948+0000 7f9c4ff42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:38:39.954 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:39.948+0000 7f9c4ff42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:38:39.955 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:39.948+0000 7f9c4ff42640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:38:39.955 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:39.948+0000 7f9c4ff42640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:38:39.955 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:39.948+0000 7f9c4ff42640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:38:40.024 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:40.016+0000 7f9c4ff42640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T06:38:46.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T06:38:46.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:38:46.503 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:38:46.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:38:46.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:38:46.891 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:38:46.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:38:46.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:38:46.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:38:46.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:38:46.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:38:46.923 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:46.916+0000 7f31d6f24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:38:46.959 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:46.952+0000 7f31d6f24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:38:46.960 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:46.956+0000 7f31d6f24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:38:47.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:38:47.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:38:47.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:38:47.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:38:47.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:38:47.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:47.226 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:38:47.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:38:47.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:47.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:38:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:47.509 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:47.504+0000 7f31d6f24640 -1 Falling back to public interface 2024-02-05T06:38:48.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:48.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:48.510 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:38:48.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:38:48.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:48.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:38:48.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:49.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:49.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:49.818 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:38:49.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:38:49.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:49.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:38:50.119 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:38:50.111+0000 7f31d6f24640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:38:50.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:51.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:51.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:51.131 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:38:51.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:38:51.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:51.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:38:51.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:52.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:52.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:52.490 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:38:52.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:38:52.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:52.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:38:52.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:53.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:53.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:53.812 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:38:53.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:38:53.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:53.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:38:54.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:55.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:55.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:55.267 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:38:55.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:38:55.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:55.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:38:55.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:56.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:56.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:56.585 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:38:56.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:38:56.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:56.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:38:56.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:57.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:57.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:57.983 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:38:57.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:38:57.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:57.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:38:58.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:38:59.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:38:59.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:38:59.304 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:38:59.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:38:59.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:38:59.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:38:59.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:00.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:00.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:00.623 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:39:00.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:39:00.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:00.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:39:01.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:02.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:02.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:02.053 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:39:02.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:39:02.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:02.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:39:02.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:03.371 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:39:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:39:03.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:39:03.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:03.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:04.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:04.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:04.678 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:39:04.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:39:04.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:04.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:39:05.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:06.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:06.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:06.335 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:39:06.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:39:06.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:06.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:39:06.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:07.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:07.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:07.956 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:39:07.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:39:07.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:07.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:39:08.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:09.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:09.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:09.335 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:39:09.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:39:09.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:09.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:39:10.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:11.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:11.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:11.186 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:39:11.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:39:11.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:11.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:39:11.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:12.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:12.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:12.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:39:12.987 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:39:12.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:12.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:39:13.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:14.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:14.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:14.406 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:39:14.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:39:14.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:14.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:39:14.744 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1140129605,v1:127.0.0.1:6811/1140129605] [v2:127.0.0.1:6812/1140129605,v1:127.0.0.1:6813/1140129605] exists,up 74a0d2be-74f2-41da-b18b-b8b00642c2b3 2024-02-05T06:39:14.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:39:14.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:39:14.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:39:14.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:39:14.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2024-02-05T06:39:14.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:39:14.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:39:14.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T06:39:14.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:39:14.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:39:14.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:39:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:39:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:39:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:39:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:39:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:39:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:39:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:39:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:39:14.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:39:14.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:39:14.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:39:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:39:14.748 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:39:14.749 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:39:14.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:39:14.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:39:14.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:39:14.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:39:14.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:39:14.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:39:14.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:39:14.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:39:14.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:39:14.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:39:14.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:39:14.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:39:14.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:39:14.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:39:14.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:39:14.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=59af7284-e6b1-4be8-a0dd-63042d888525 2024-02-05T06:39:14.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 59af7284-e6b1-4be8-a0dd-63042d888525' 2024-02-05T06:39:14.754 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 59af7284-e6b1-4be8-a0dd-63042d888525 2024-02-05T06:39:14.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:39:14.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCSgsBlyeOwLRAABf5zaut4xRZZi/hbuOgsYg== 2024-02-05T06:39:14.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCSgsBlyeOwLRAABf5zaut4xRZZi/hbuOgsYg=="}' 2024-02-05T06:39:14.770 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 59af7284-e6b1-4be8-a0dd-63042d888525 -i td/osd-scrub-repair/2/new.json 2024-02-05T06:39:15.128 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:39:15.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T06:39:15.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCSgsBlyeOwLRAABf5zaut4xRZZi/hbuOgsYg== --osd-uuid 59af7284-e6b1-4be8-a0dd-63042d888525 2024-02-05T06:39:15.180 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:15.171+0000 7f12c46ee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:39:15.222 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:15.215+0000 7f12c46ee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:39:15.224 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:15.215+0000 7f12c46ee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:39:15.224 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:15.219+0000 7f12c46ee640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:39:15.224 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:15.219+0000 7f12c46ee640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:39:15.224 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:15.219+0000 7f12c46ee640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:39:15.313 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:15.307+0000 7f12c46ee640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T06:39:21.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T06:39:21.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:39:21.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T06:39:21.995 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T06:39:21.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:39:22.381 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T06:39:22.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T06:39:22.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:39:22.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:39:22.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:39:22.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:39:22.413 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:22.407+0000 7f4c05140640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:39:22.445 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:22.439+0000 7f4c05140640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:39:22.446 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:22.439+0000 7f4c05140640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:39:22.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T06:39:22.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:39:22.714 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:39:22.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:39:22.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:39:22.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:39:22.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:22.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:39:22.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:22.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:23.002 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:22.995+0000 7f4c05140640 -1 Falling back to public interface 2024-02-05T06:39:23.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:24.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:24.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:24.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:39:24.038 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:39:24.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:24.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:24.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:25.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:25.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:25.349 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:39:25.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:39:25.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:25.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:25.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:25.683 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:25.675+0000 7f4c05140640 -1 osd.2 0 log_to_monitors true 2024-02-05T06:39:26.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:26.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:26.661 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:39:26.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:39:26.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:26.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:26.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:27.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:27.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:27.983 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:39:27.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:39:27.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:27.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:28.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:29.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:29.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:29.324 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:39:29.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:39:29.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:29.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:29.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:30.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:30.723 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:39:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:39:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:30.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:31.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:32.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:32.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:32.041 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:39:32.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:39:32.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:32.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:32.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:33.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:33.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:33.373 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:39:33.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:39:33.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:33.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:33.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:34.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:34.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:34.830 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:39:34.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:39:34.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:34.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:35.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:36.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:36.189 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:36.190 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:39:36.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:39:36.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:36.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:36.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:37.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:37.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:37.591 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:39:37.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:39:37.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:37.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:37.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:38.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:38.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:38.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:39:38.988 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:39:38.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:38.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:39.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:40.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:40.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:40.386 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:39:40.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:39:40.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:40.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:41.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:42.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:42.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:42.194 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:39:42.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:39:42.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:42.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:42.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:43.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:43.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:43.632 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:39:43.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:39:43.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:43.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:44.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:45.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:45.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:45.418 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:39:45.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:39:45.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:45.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:45.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:46.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:46.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:46.929 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:39:46.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:39:46.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:46.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:47.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:48.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:48.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:48.349 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:39:48.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:39:48.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:48.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:49.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:50.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:50.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:50.178 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:39:50.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:39:50.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:50.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:39:50.521 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3776823108,v1:127.0.0.1:6819/3776823108] [v2:127.0.0.1:6820/3776823108,v1:127.0.0.1:6821/3776823108] exists,up 59af7284-e6b1-4be8-a0dd-63042d888525 2024-02-05T06:39:50.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:39:50.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:39:50.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:39:50.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:39:50.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2024-02-05T06:39:50.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:39:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:39:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T06:39:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:39:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T06:39:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:39:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:39:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:39:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:39:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T06:39:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T06:39:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:39:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:39:50.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:39:50.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:39:50.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:39:50.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:39:50.526 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:39:50.526 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:39:50.526 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:39:50.527 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:39:50.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:39:50.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:39:50.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:39:50.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:39:50.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:39:50.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:39:50.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:39:50.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:39:50.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:39:50.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:39:50.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:39:50.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:39:50.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T06:39:50.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:39:50.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d6827d7b-3c2b-4b0a-9b53-cfbec8ed5909 2024-02-05T06:39:50.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 d6827d7b-3c2b-4b0a-9b53-cfbec8ed5909' 2024-02-05T06:39:50.529 INFO:tasks.workunit.client.0.smithi148.stdout:add osd3 d6827d7b-3c2b-4b0a-9b53-cfbec8ed5909 2024-02-05T06:39:50.529 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:39:50.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2gsBlMPNNIBAAfTb0EyWnv6+7xzlE+Mq60w== 2024-02-05T06:39:50.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2gsBlMPNNIBAAfTb0EyWnv6+7xzlE+Mq60w=="}' 2024-02-05T06:39:50.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d6827d7b-3c2b-4b0a-9b53-cfbec8ed5909 -i td/osd-scrub-repair/3/new.json 2024-02-05T06:39:51.055 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:39:51.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-02-05T06:39:51.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2gsBlMPNNIBAAfTb0EyWnv6+7xzlE+Mq60w== --osd-uuid d6827d7b-3c2b-4b0a-9b53-cfbec8ed5909 2024-02-05T06:39:51.110 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:51.102+0000 7fe284e3b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:39:51.169 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:51.162+0000 7fe284e3b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:39:51.171 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:51.162+0000 7fe284e3b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:39:51.171 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:51.162+0000 7fe284e3b640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T06:39:51.172 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:51.162+0000 7fe284e3b640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T06:39:51.172 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:51.162+0000 7fe284e3b640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T06:39:51.223 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:51.218+0000 7fe284e3b640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-02-05T06:39:57.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-02-05T06:39:57.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:39:57.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T06:39:57.648 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd3 key to auth repository 2024-02-05T06:39:57.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:39:58.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T06:39:58.092 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.3 2024-02-05T06:39:58.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:39:58.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:39:58.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:39:58.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:39:58.125 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:58.122+0000 7fe82244e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:39:58.158 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:58.154+0000 7fe82244e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:39:58.160 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:58.154+0000 7fe82244e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:39:58.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T06:39:58.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:39:58.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T06:39:58.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:39:58.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:39:58.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:58.419 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:39:58.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:39:58.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:58.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:39:58.715 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:39:58.710+0000 7fe82244e640 -1 Falling back to public interface 2024-02-05T06:39:58.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:39:59.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:39:59.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:39:59.768 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:39:59.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:39:59.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:39:59.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:00.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:01.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:01.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:01.085 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:40:01.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:40:01.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:01.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:01.375 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:40:01.370+0000 7fe82244e640 -1 osd.3 0 log_to_monitors true 2024-02-05T06:40:01.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:02.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:02.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:02.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:40:02.400 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:40:02.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:02.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:02.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:03.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:03.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:03.737 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:40:03.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:40:03.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:03.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:04.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:05.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:05.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:05.057 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:40:05.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:40:05.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:05.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:05.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:06.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:06.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:06.428 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:40:06.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:40:06.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:06.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:06.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:07.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:07.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:07.729 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:40:07.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:40:07.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:07.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:08.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:09.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:09.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:09.043 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:40:09.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:40:09.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:09.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:09.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:10.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:10.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:10.446 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:40:10.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:40:10.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:10.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:10.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:11.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:11.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:11.753 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:40:11.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:40:11.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:11.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:12.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:13.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:13.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:13.153 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:40:13.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:40:13.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:13.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:13.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:14.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:14.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:14.622 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:40:14.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:40:14.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:14.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:14.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:15.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:15.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:15.940 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:40:15.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:40:15.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:15.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:16.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:17.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:17.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:17.776 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:40:17.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:40:17.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:17.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:18.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:19.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:19.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:19.513 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:40:19.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:40:19.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:19.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:20.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:21.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:21.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:21.588 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:40:21.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:40:21.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:21.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:22.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:23.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:23.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:23.390 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:40:23.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:40:23.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:23.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:23.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:24.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:24.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:24.868 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:40:24.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:40:24.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:24.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:25.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:26.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:26.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:26.279 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:40:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:40:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:26.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:26.595 INFO:tasks.workunit.client.0.smithi148.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3915182073,v1:127.0.0.1:6827/3915182073] [v2:127.0.0.1:6828/3915182073,v1:127.0.0.1:6829/3915182073] exists,up d6827d7b-3c2b-4b0a-9b53-cfbec8ed5909 2024-02-05T06:40:26.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:40:26.596 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:40:26.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:40:26.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: create_ec_pool ecpool true k=2 m=2 2024-02-05T06:40:26.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T06:40:26.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T06:40:26.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-02-05T06:40:26.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T06:40:26.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-02-05T06:40:27.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T06:40:27.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T06:40:27.884 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'ecpool' created 2024-02-05T06:40:27.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:40:28.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-02-05T06:40:28.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-02-05T06:40:29.365 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 1 allow_ec_overwrites to true 2024-02-05T06:40:29.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T06:40:29.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:40:29.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:40:29.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:40:29.383 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:40:29.384 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:40:29.384 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:40:29.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:40:29.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:40:29.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:40:29.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:40:29.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:40:29.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:40:29.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:40:29.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:40:29.508 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:40:29.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:40:29.820 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:40:29.820 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:40:29.820 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T06:40:29.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:40:29.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:29.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:40:29.968 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-02-05T06:40:29.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-02-05T06:40:29.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-02-05T06:40:29.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:29.969 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:40:30.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-02-05T06:40:30.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-02-05T06:40:30.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672982' 2024-02-05T06:40:30.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:30.130 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:40:30.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-02-05T06:40:30.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-02-05T06:40:30.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672982 2-64424509453' 2024-02-05T06:40:30.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:30.265 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:40:30.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345927 2024-02-05T06:40:30.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345927 2024-02-05T06:40:30.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672982 2-64424509453 3-85899345927' 2024-02-05T06:40:30.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:30.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-02-05T06:40:30.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:30.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:40:30.399 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-02-05T06:40:30.399 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:30.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-02-05T06:40:30.400 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836507 2024-02-05T06:40:30.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-02-05T06:40:30.400 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:40:30.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-02-05T06:40:30.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:40:31.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:40:31.684 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:40:32.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-02-05T06:40:32.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:32.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-02-05T06:40:32.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:32.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:40:32.007 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-02-05T06:40:32.007 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:32.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-02-05T06:40:32.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-02-05T06:40:32.009 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672982 2024-02-05T06:40:32.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:40:32.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-02-05T06:40:32.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:32.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-02-05T06:40:32.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:32.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:40:32.343 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-02-05T06:40:32.343 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:32.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-02-05T06:40:32.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-02-05T06:40:32.344 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509453 2024-02-05T06:40:32.345 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:40:32.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-02-05T06:40:32.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:32.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345927 2024-02-05T06:40:32.686 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:32.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:40:32.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:32.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345927 2024-02-05T06:40:32.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345927 2024-02-05T06:40:32.691 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.3 seq 85899345927 2024-02-05T06:40:32.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2024-02-05T06:40:32.691 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:40:33.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345928 -lt 85899345927 2024-02-05T06:40:33.025 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:40:33.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:33.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:33.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:40:33.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:40:33.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:40:33.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:40:33.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:40:33.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:40:33.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:40:33.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:40:33.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:40:33.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:40:33.808 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:33.808 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:34.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:40:34.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:40:34.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:40:34.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T06:40:34.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-02-05T06:40:34.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:40:34.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:40:34.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T06:40:34.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:40:34.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:40:34.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:40:34.587 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:40:34.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:40:35.106 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:40:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:40:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:40:35.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T06:40:35.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: get_primary ecpool SOMETHING 2024-02-05T06:40:35.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T06:40:35.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T06:40:35.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T06:40:35.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:40:35.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: local primary=1 2024-02-05T06:40:35.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: get_osds ecpool SOMETHING 2024-02-05T06:40:35.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: sed -e s/1// 2024-02-05T06:40:35.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T06:40:35.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-02-05T06:40:35.779 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-02-05T06:40:35.779 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:40:36.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T06:40:36.114 INFO:tasks.workunit.client.0.smithi148.stderr:0 2024-02-05T06:40:36.114 INFO:tasks.workunit.client.0.smithi148.stderr:2 2024-02-05T06:40:36.115 INFO:tasks.workunit.client.0.smithi148.stderr:3' 2024-02-05T06:40:36.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 3 2024-02-05T06:40:36.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: osds=('0' '2' '3') 2024-02-05T06:40:36.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: local -a osds 2024-02-05T06:40:36.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: local not_primary_first=0 2024-02-05T06:40:36.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: local not_primary_second=2 2024-02-05T06:40:36.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:871: unfound_erasure_coded: local not_primary_third=3 2024-02-05T06:40:36.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: pids= 2024-02-05T06:40:36.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:40:36.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T06:40:36.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T06:40:36.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 474384"' 2024-02-05T06:40:36.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 474384' 2024-02-05T06:40:36.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:40:36.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T06:40:36.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T06:40:36.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 474385"' 2024-02-05T06:40:36.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 474385' 2024-02-05T06:40:36.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:40:36.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T06:40:36.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T06:40:36.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/474387: /' 2024-02-05T06:40:36.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:40:36.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 474388"' 2024-02-05T06:40:36.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 474388' 2024-02-05T06:40:36.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: wait_background pids 2024-02-05T06:40:36.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 474384 474385 474388' 2024-02-05T06:40:36.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T06:40:36.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T06:40:36.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 474384 2024-02-05T06:40:36.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:40:36.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/474390: /' 2024-02-05T06:40:36.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:40:36.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/474392: /' 2024-02-05T06:40:39.243 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:40:39.244 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:40:39.244 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T06:40:39.244 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:40:39.244 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:40:39.244 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:40:39.244 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:40:39.244 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T06:40:39.244 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:40:39.244 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T06:40:39.244 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:40:39.244 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:40:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:40:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:40:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:40:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:40:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:40:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:40:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:40:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T06:40:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:40:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:40:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-02-05T06:40:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:474390: remove 2#1:eb822e21:::SOMETHING:head# 2024-02-05T06:40:39.246 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T06:40:39.246 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:40:39.246 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:40:39.246 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T06:40:39.246 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:40:39.246 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:40:39.246 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:40:39.246 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:40:39.251 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:40:39.251 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:40:39.251 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:40:39.251 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:40:39.251 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:40:39.251 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:40:39.251 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:40:39.251 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:40:39.252 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:40:39.252 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:40:39.252 INFO:tasks.workunit.client.0.smithi148.stderr:474390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:40:39.252 INFO:tasks.workunit.client.0.smithi148.stderr:474390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:40:39.252 INFO:tasks.workunit.client.0.smithi148.stderr:474390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:40:39.252 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:40:39.252 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:40:39.252 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:40:39.252 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:40:39.252 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:40:39.253 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:40:39.253 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:40:39.253 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:40:39.253 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:40:39.253 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:40:39.253 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:40:39.253 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:40:39.253 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T06:40:39.253 INFO:tasks.workunit.client.0.smithi148.stderr:474390: start osd.2 2024-02-05T06:40:39.253 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:40:39.297 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:40:39.298 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:40:39.298 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T06:40:39.298 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:40:39.298 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:40:39.298 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:40:39.298 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:40:39.298 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T06:40:39.298 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:40:39.298 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T06:40:39.298 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:40:39.298 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:40:39.298 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:40:39.298 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: remove 3#1:eb822e21:::SOMETHING:head# 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T06:40:39.299 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:40:39.300 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T06:40:39.300 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:40:39.300 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:40:39.301 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:40:39.301 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:40:39.301 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T06:40:39.301 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T06:40:39.301 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:40:39.301 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:40:39.302 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:40:39.303 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:40:39.303 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:40:39.303 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:40:39.303 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T06:40:39.304 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T06:40:39.304 INFO:tasks.workunit.client.0.smithi148.stderr:474392: start osd.3 2024-02-05T06:40:39.304 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:40:39.322 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:40:39.322 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:40:39.322 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:40:39.322 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:40:39.322 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:40:39.322 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:40:39.322 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:40:39.322 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:40:39.322 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:40:39.322 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:40:39.322 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:40:39.322 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:40:39.323 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:40:39.323 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:40:39.323 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:40:39.323 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:40:39.323 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:40:39.323 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:40:39.323 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:40:39.323 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:40:39.323 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:40:39.323 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:40:39.323 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-02-05T06:40:39.323 INFO:tasks.workunit.client.0.smithi148.stderr:474387: remove 1#1:eb822e21:::SOMETHING:head# 2024-02-05T06:40:39.323 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:40:39.324 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:40:39.324 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:40:39.324 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:40:39.324 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:40:39.324 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:40:39.324 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:40:39.324 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:40:39.326 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:40:39.326 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:40:39.326 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:40:39.326 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:40:39.327 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:40:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:40:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:40:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:40:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:40:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:40:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:40:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:40:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:474387: start osd.0 2024-02-05T06:40:39.328 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8650514f-daba-4122-8186-14969aa341bf --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61355/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:40:43.976 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T06:40:43.977 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T06:40:43.977 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:40:43.977 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:40:43.977 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:40:43.977 INFO:tasks.workunit.client.0.smithi148.stderr:474390: 2024-02-05T06:40:39.277+0000 7fd8c71ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:40:43.977 INFO:tasks.workunit.client.0.smithi148.stderr:474390: 2024-02-05T06:40:39.281+0000 7fd8c71ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:40:43.977 INFO:tasks.workunit.client.0.smithi148.stderr:474390: 2024-02-05T06:40:39.281+0000 7fd8c71ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:40:43.977 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T06:40:43.977 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:40:43.977 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:40:43.977 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:40:43.977 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:40:43.977 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:43.977 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:40:43.978 INFO:tasks.workunit.client.0.smithi148.stderr:474390: 0 2024-02-05T06:40:43.978 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:43.978 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:40:43.978 INFO:tasks.workunit.client.0.smithi148.stderr:474390: 2024-02-05T06:40:39.841+0000 7fd8c71ed640 -1 Falling back to public interface 2024-02-05T06:40:43.978 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:43.978 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:43.978 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:43.978 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:40:43.978 INFO:tasks.workunit.client.0.smithi148.stderr:474390: 1 2024-02-05T06:40:43.978 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:43.978 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:40:43.978 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:43.978 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:43.978 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:43.979 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:40:43.979 INFO:tasks.workunit.client.0.smithi148.stderr:474390: 2 2024-02-05T06:40:43.979 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:43.979 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:40:43.979 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:43.979 INFO:tasks.workunit.client.0.smithi148.stderr:474390: 2024-02-05T06:40:42.925+0000 7fd8c71ed640 -1 osd.2 29 log_to_monitors true 2024-02-05T06:40:43.979 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:43.979 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:43.979 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:40:43.979 INFO:tasks.workunit.client.0.smithi148.stderr:474390: 3 2024-02-05T06:40:43.979 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:43.979 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:40:44.143 INFO:tasks.workunit.client.0.smithi148.stderr:474390: osd.2 up in weight 1 up_from 34 up_thru 0 down_at 30 last_c474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:40:44.144 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:40:44.144 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:40:44.144 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:40:44.144 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:40:44.144 INFO:tasks.workunit.client.0.smithi148.stderr:474387: 2024-02-05T06:40:39.365+0000 7fc8b2cb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:40:44.144 INFO:tasks.workunit.client.0.smithi148.stderr:474387: 2024-02-05T06:40:39.369+0000 7fc8b2cb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:40:44.144 INFO:tasks.workunit.client.0.smithi148.stderr:474387: 2024-02-05T06:40:39.369+0000 7fc8b2cb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:40:44.144 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:40:44.144 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:40:44.144 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:40:44.144 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:40:44.145 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:40:44.145 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:44.145 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:40:44.145 INFO:tasks.workunit.client.0.smithi148.stderr:474387: 0 2024-02-05T06:40:44.145 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:44.145 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:40:44.145 INFO:tasks.workunit.client.0.smithi148.stderr:474387: 2024-02-05T06:40:39.941+0000 7fc8b2cb6640 -1 Falling back to public interface 2024-02-05T06:40:44.145 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:44.145 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:44.145 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:44.145 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:40:44.145 INFO:tasks.workunit.client.0.smithi148.stderr:474387: 1 2024-02-05T06:40:44.145 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:44.145 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:40:44.146 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:44.146 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:44.146 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:44.146 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:40:44.146 INFO:tasks.workunit.client.0.smithi148.stderr:474387: 2 2024-02-05T06:40:44.146 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:44.146 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:40:44.146 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:44.146 INFO:tasks.workunit.client.0.smithi148.stderr:474387: 2024-02-05T06:40:43.281+0000 7fc8b2cb6640 -1 osd.0 29 log_to_monitors true 2024-02-05T06:40:44.146 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:44.146 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:44.146 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:40:44.146 INFO:tasks.workunit.client.0.smithi148.stderr:474387: 3 2024-02-05T06:40:44.147 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:44.147 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:40:44.147 INFO:tasks.workunit.client.0.smithi148.stderr:474387: osd.0 up in weight 1 up_from 34 up_thru 0 down_at 30 last_c474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T06:40:44.147 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T06:40:44.147 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:40:44.147 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:40:44.147 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:40:44.147 INFO:tasks.workunit.client.0.smithi148.stderr:474392: 2024-02-05T06:40:39.325+0000 7fec9c160640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:40:44.147 INFO:tasks.workunit.client.0.smithi148.stderr:474392: 2024-02-05T06:40:39.329+0000 7fec9c160640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:40:44.147 INFO:tasks.workunit.client.0.smithi148.stderr:474392: 2024-02-05T06:40:39.329+0000 7fec9c160640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:40:44.147 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T06:40:44.147 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:40:44.147 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T06:40:44.148 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:40:44.148 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:40:44.148 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:44.148 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:40:44.148 INFO:tasks.workunit.client.0.smithi148.stderr:474392: 0 2024-02-05T06:40:44.148 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:44.148 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:44.148 INFO:tasks.workunit.client.0.smithi148.stderr:474392: 2024-02-05T06:40:39.893+0000 7fec9c160640 -1 Falling back to public interface 2024-02-05T06:40:44.148 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:44.148 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:44.148 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:44.148 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:40:44.148 INFO:tasks.workunit.client.0.smithi148.stderr:474392: 1 2024-02-05T06:40:44.148 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:44.149 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:44.149 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:44.149 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:44.149 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:44.149 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:40:44.149 INFO:tasks.workunit.client.0.smithi148.stderr:474392: 2 2024-02-05T06:40:44.149 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:44.149 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:44.149 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:40:44.149 INFO:tasks.workunit.client.0.smithi148.stderr:474392: 2024-02-05T06:40:43.161+0000 7fec9c160640 -1 osd.3 29 log_to_monitors true 2024-02-05T06:40:44.149 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:40:44.149 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:40:44.149 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:40:44.149 INFO:tasks.workunit.client.0.smithi148.stderr:474392: 3 2024-02-05T06:40:44.150 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:40:44.150 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:40:44.773 INFO:tasks.workunit.client.0.smithi148.stderr:474392: osd.3 up in weight 1 up_from 34 up_thru 0 down_at 30 last_clean_interval [15,29) [v2:127.0.0.1:6802/3399996017,v1:127.0.0.1:6803/3399996017] [v2:127.0.0.1:6804/3399996017,v1:127.0.0.1:6805/3399996017] exists,up 59af7284-e6b1-4be8-a0dd-63042d888525 2024-02-05T06:40:44.774 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:40:44.774 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:40:44.774 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:40:44.774 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:40:44.774 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:40:44.774 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:40:44.774 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:40:44.774 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:40:44.774 INFO:tasks.workunit.client.0.smithi148.stderr:474390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:40:44.774 INFO:tasks.workunit.client.0.smithi148.stderr:474390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:40:44.774 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:40:44.774 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:40:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:40:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:40:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:40:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:40:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:40:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:40:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:40:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:40:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:474390: 1 2024-02-05T06:40:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:474390: 2 2024-02-05T06:40:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:474390: 3' 2024-02-05T06:40:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:40:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:40:44.776 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-02-05T06:40:44.776 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-02-05T06:40:44.776 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066' 2024-02-05T06:40:44.776 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:44.776 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:40:44.776 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-02-05T06:40:44.776 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-02-05T06:40:44.776 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-42949672986' 2024-02-05T06:40:44.776 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:44.960 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph lean_interval [20,29) [v2:127.0.0.1:6818/885317945,v1:127.0.0.1:6819/885317945] [v2:127.0.0.1:6820/885317945,v1:127.0.0.1:6821/885317945] exists,up d6827d7b-3c2b-4b0a-9b53-cfbec8ed5909 2024-02-05T06:40:44.960 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:40:44.960 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:40:44.960 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:40:44.960 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:40:44.960 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:40:44.960 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:40:44.960 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: 1 2024-02-05T06:40:44.961 INFO:tasks.workunit.client.0.smithi148.stderr:474392: 2 2024-02-05T06:40:44.962 INFO:tasks.workunit.client.0.smithi148.stderr:474392: 3' 2024-02-05T06:40:44.962 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:40:44.962 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:44.962 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:40:44.962 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888068 2024-02-05T06:40:44.962 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888068 2024-02-05T06:40:44.962 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888068' 2024-02-05T06:40:44.962 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:44.962 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:40:44.962 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672987 2024-02-05T06:40:44.962 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672987 2024-02-05T06:40:44.962 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888068 1-42949672987' 2024-02-05T06:40:44.962 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:44.963 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph telllean_interval [5,29) [v2:127.0.0.1:6826/3063623999,v1:127.0.0.1:6827/3063623999] [v2:127.0.0.1:6828/3063623999,v1:127.0.0.1:6829/3063623999] exists,up 7e7dc194-3d95-40f8-8258-f3a08cc6a777 2024-02-05T06:40:44.963 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:40:44.963 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:40:44.963 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:40:44.963 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:40:44.964 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:40:44.964 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:40:44.964 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:40:44.964 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:40:44.964 INFO:tasks.workunit.client.0.smithi148.stderr:474387: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:40:44.964 INFO:tasks.workunit.client.0.smithi148.stderr:474387: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:40:44.964 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:40:44.964 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:40:44.964 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:40:44.964 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:40:44.964 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:40:44.964 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:40:44.964 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: 1 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: 2 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: 3' 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888067 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888067 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888067' 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:40:44.965 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672988 2024-02-05T06:40:44.966 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672988 2024-02-05T06:40:44.966 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888067 1-42949672988' 2024-02-05T06:40:44.966 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:45.833 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph ttell osd.2 flush_pg_stats 2024-02-05T06:40:45.833 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-02-05T06:40:45.833 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-02-05T06:40:45.833 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-42949672986 2-146028888066' 2024-02-05T06:40:45.833 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:45.833 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:40:45.833 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-02-05T06:40:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-02-05T06:40:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-42949672986 2-146028888066 3-146028888066' 2024-02-05T06:40:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888066 2024-02-05T06:40:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:40:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888066 2024-02-05T06:40:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-02-05T06:40:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-02-05T06:40:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:474390: waiting osd.0 seq 146028888066 2024-02-05T06:40:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:40:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888066 2024-02-05T06:40:45.834 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: waiting osd.1 seq 42949672986 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672986 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888066 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:40:45.835 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888066 2024-02-05T06:40:45.971 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephell osd.2 flush_pg_stats 2024-02-05T06:40:45.971 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888068 2024-02-05T06:40:45.971 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888068 2024-02-05T06:40:45.972 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888067 1-42949672988 2-146028888068' 2024-02-05T06:40:45.972 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:45.972 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:40:45.972 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888067 2024-02-05T06:40:45.972 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888067 2024-02-05T06:40:45.972 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888067 1-42949672988 2-146028888068 3-146028888067' 2024-02-05T06:40:45.972 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:45.972 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888067 2024-02-05T06:40:45.972 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:45.972 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:40:45.972 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888067 2024-02-05T06:40:45.972 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:45.972 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888067 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: waiting osd.0 seq 146028888067 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888067 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672988 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672988 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672988 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: waiting osd.1 seq 42949672988 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672988 2024-02-05T06:40:45.973 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:45.974 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888068 2024-02-05T06:40:45.974 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:45.974 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:40:45.974 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888068 2024-02-05T06:40:45.985 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cepht osd.2 flush_pg_stats 2024-02-05T06:40:45.985 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888067 2024-02-05T06:40:45.985 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888067 2024-02-05T06:40:45.985 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888068 1-42949672987 2-146028888067' 2024-02-05T06:40:45.985 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:40:45.985 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:40:45.985 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888068 2024-02-05T06:40:45.985 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888068 2024-02-05T06:40:45.985 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888068 1-42949672987 2-146028888067 3-146028888068' 2024-02-05T06:40:45.985 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:45.985 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888068 2024-02-05T06:40:45.985 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:45.985 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:40:45.985 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888068 2024-02-05T06:40:45.986 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:45.986 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888068 2024-02-05T06:40:45.986 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888068' 2024-02-05T06:40:45.986 INFO:tasks.workunit.client.0.smithi148.stderr:474392: waiting osd.0 seq 146028888068 2024-02-05T06:40:45.986 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:40:45.986 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888068 2024-02-05T06:40:45.986 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:45.986 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672987 2024-02-05T06:40:45.986 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:45.986 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:40:45.986 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672987 2024-02-05T06:40:45.986 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:45.986 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672987 2024-02-05T06:40:45.986 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-02-05T06:40:45.987 INFO:tasks.workunit.client.0.smithi148.stderr:474392: waiting osd.1 seq 42949672987 2024-02-05T06:40:45.987 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:40:45.987 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672987 2024-02-05T06:40:45.987 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:45.987 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888067 2024-02-05T06:40:45.987 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:45.987 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:40:45.987 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888067 2024-02-05T06:40:47.505 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtesttest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:47.506 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-02-05T06:40:47.506 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888066' 2024-02-05T06:40:47.506 INFO:tasks.workunit.client.0.smithi148.stderr:474390: waiting osd.2 seq 146028888066 2024-02-05T06:40:47.506 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:40:47.506 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888066 2024-02-05T06:40:47.506 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:47.506 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-146028888066 2024-02-05T06:40:47.506 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:47.506 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:40:47.506 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-146028888066 2024-02-05T06:40:47.506 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:47.506 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-02-05T06:40:47.506 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 146028888066' 2024-02-05T06:40:47.506 INFO:tasks.workunit.client.0.smithi148.stderr:474390: waiting osd.3 seq 146028888066 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888066 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:40:47.507 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:40:47.508 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:47.508 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:47.508 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:40:47.508 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-05T06:40:48.726 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalonest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:48.726 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888068 2024-02-05T06:40:48.726 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888068' 2024-02-05T06:40:48.726 INFO:tasks.workunit.client.0.smithi148.stderr:474387: waiting osd.2 seq 146028888068 2024-02-05T06:40:48.727 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:40:48.727 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888068 2024-02-05T06:40:48.727 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:48.727 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-146028888067 2024-02-05T06:40:48.727 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:48.727 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:40:48.727 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-146028888067 2024-02-05T06:40:48.727 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:48.727 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888067 2024-02-05T06:40:48.727 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 146028888067' 2024-02-05T06:40:48.727 INFO:tasks.workunit.client.0.smithi148.stderr:474387: waiting osd.3 seq 146028888067 2024-02-05T06:40:48.727 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:40:48.727 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888067 2024-02-05T06:40:48.727 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:40:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:40:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:40:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888067 2024-02-05T06:40:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:40:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:40:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:40:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:40:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:40:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:40:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:40:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:40:48.729 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:40:48.729 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:40:48.729 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:40:48.776 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone./clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:48.776 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888067 2024-02-05T06:40:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2024-02-05T06:40:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:474392: waiting osd.2 seq 146028888067 2024-02-05T06:40:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:40:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888067 2024-02-05T06:40:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:40:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-146028888068 2024-02-05T06:40:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:40:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:40:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-146028888068 2024-02-05T06:40:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:40:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888068 2024-02-05T06:40:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 146028888068' 2024-02-05T06:40:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:474392: waiting osd.3 seq 146028888068 2024-02-05T06:40:48.777 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:40:48.778 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888068 2024-02-05T06:40:48.778 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:40:48.778 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:40:48.778 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:40:48.778 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888068 2024-02-05T06:40:48.778 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:40:48.778 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:48.778 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:48.778 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:40:48.778 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:40:48.778 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:40:48.778 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:40:48.778 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:40:48.778 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:40:48.779 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:40:48.779 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:40:48.779 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:40:48.779 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:40:48.898 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.clie/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:40:48.899 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-05T06:40:48.899 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:40:48.899 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:40:48.899 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:40:48.899 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:40:48.899 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:40:48.899 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:40:48.899 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:40:48.899 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:40:48.899 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:40:48.899 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:40:48.899 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:40:48.899 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:40:48.900 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:48.900 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:48.900 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:40:48.900 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T06:40:48.900 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:40:48.900 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:40:48.900 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:40:48.900 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:40:48.900 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:40:48.900 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:40:48.900 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:40:48.900 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:40:48.900 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:40:48.900 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:40:48.901 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:40:48.901 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:40:48.901 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:40:50.259 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:50.259 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:50.259 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:40:50.259 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-05T06:40:50.259 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:40:50.259 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-05T06:40:50.259 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:40:50.259 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:40:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:40:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:40:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:40:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:40:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:40:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:40:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:40:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:40:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:40:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:40:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:50.260 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:40:50.261 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T06:40:50.261 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:40:50.261 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:40:50.261 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:40:50.261 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:40:50.261 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:40:50.261 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:40:50.261 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:40:50.261 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:40:50.261 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:40:50.341 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtestent.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:50.341 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:50.341 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:40:50.341 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-05T06:40:50.342 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:40:50.342 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-05T06:40:50.342 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:40:50.342 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:40:50.342 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:40:50.342 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:40:50.342 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:40:50.342 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:40:50.342 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:40:50.342 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:40:50.343 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:40:50.344 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:40:50.344 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:40:50.361 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/cl653: wait_for_clean: true 2024-02-05T06:40:50.361 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:40:50.361 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:40:50.361 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:40:50.361 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:40:50.361 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:40:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:40:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:40:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:40:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:40:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T06:40:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:40:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:40:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:40:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:40:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:40:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:40:50.362 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:40:50.363 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:40:50.363 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:40:50.363 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T06:40:50.363 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:40:50.363 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T06:40:50.363 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:40:50.363 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:40:50.363 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:40:50.363 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:40:50.363 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:40:51.902 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:40:51.903 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:40:51.903 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:40:51.903 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:40:51.903 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:40:51.903 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:40:51.903 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:40:51.903 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:40:51.903 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:40:51.903 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:40:51.903 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:40:51.903 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:40:51.903 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:40:51.904 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:51.904 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:51.904 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:40:51.904 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T06:40:51.904 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:40:51.904 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:40:51.904 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:40:51.904 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:40:51.904 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:40:51.904 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:40:51.904 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:40:51.905 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:40:51.905 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:40:51.905 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T06:40:51.905 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:40:51.905 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T06:40:51.905 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:40:51.905 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:40:52.007 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/stanone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T06:40:52.007 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:40:52.008 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T06:40:52.008 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:40:52.008 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:40:52.008 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:40:52.008 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:40:52.008 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:40:52.008 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:40:52.008 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:40:52.008 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:40:52.008 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:40:52.008 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:40:52.008 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:52.008 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:52.008 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:40:52.009 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:40:52.537 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalselect(contains("stale") | not)' 2024-02-05T06:40:52.537 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:40:52.537 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:40:52.537 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:40:52.537 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:40:52.538 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:52.538 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:52.538 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:40:52.538 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T06:40:52.538 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T06:40:52.538 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T06:40:52.538 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T06:40:52.538 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T06:40:52.538 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T06:40:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T06:40:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T06:40:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T06:40:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T06:40:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-02-05T06:40:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:40:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-02-05T06:40:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:40:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:40:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:40:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:40:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:40:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:40:52.539 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:40:52.540 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:40:52.540 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:40:52.603 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.cliedalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:40:52.603 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:40:52.603 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:40:52.603 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:40:52.603 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:40:52.603 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:40:52.603 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:40:52.603 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:40:52.604 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:52.604 INFO:tasks.workunit.client.0.smithi148.stderr:474387: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:52.604 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:40:52.604 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:40:52.604 INFO:tasks.workunit.client.0.smithi148.stderr:474387: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:40:52.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T06:40:52.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T06:40:52.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 474385 2024-02-05T06:40:52.690 INFO:tasks.workunit.client.0.smithi148.stderr:one/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:40:52.690 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:40:52.690 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:40:52.690 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:40:52.690 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:40:52.690 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:40:52.690 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:40:52.691 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:40:52.691 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:52.691 INFO:tasks.workunit.client.0.smithi148.stderr:474392: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:52.691 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:40:52.691 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:40:52.691 INFO:tasks.workunit.client.0.smithi148.stderr:474392: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:40:52.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T06:40:52.909 INFO:tasks.workunit.client.0.smithi148.stderr:nt.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:40:52.909 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:40:52.909 INFO:tasks.workunit.client.0.smithi148.stderr:474390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:40:52.909 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:40:52.909 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:40:52.909 INFO:tasks.workunit.client.0.smithi148.stderr:474390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:40:52.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T06:40:52.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T06:40:52.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 474388 2024-02-05T06:40:52.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T06:40:52.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T06:40:52.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T06:40:52.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: return_code=0 2024-02-05T06:40:52.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:882: unfound_erasure_coded: '[' 0 -ne 0 ']' 2024-02-05T06:40:52.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: get_pg ecpool SOMETHING 2024-02-05T06:40:52.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T06:40:52.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T06:40:52.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T06:40:52.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:40:53.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: local pg=1.0 2024-02-05T06:40:53.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: unfound_erasure_coded: repair 1.0 2024-02-05T06:40:53.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-02-05T06:40:53.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-02-05T06:40:53.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:40:53.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:40:53.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:40:53.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:40:53.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T06:40:27.746330+0000 2024-02-05T06:40:53.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-02-05T06:40:53.853 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-02-05T06:40:53.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-02-05T06:40:27.746330+0000 2024-02-05T06:40:53.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:40:53.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:40:27.746330+0000 2024-02-05T06:40:53.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:40:53.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:40:53.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:40:53.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:40:53.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:40:53.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:40:53.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:40:53.870 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:40:54.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:40:27.746330+0000 '>' 2024-02-05T06:40:27.746330+0000 2024-02-05T06:40:54.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:40:55.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:40:55.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:40:55.172 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:40:55.172 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:40:55.172 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:40:55.172 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:40:55.173 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:40:55.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:40:54.396408+0000 '>' 2024-02-05T06:40:27.746330+0000 2024-02-05T06:40:55.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:40:55.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: seq 1 60 2024-02-05T06:40:55.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: for f in `seq 1 60` 2024-02-05T06:40:55.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: grep '1/1 objects unfound' 2024-02-05T06:40:55.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: ceph -s 2024-02-05T06:40:55.865 INFO:tasks.workunit.client.0.smithi148.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:40:55.865 INFO:tasks.workunit.client.0.smithi148.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:40:55.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: break 2024-02-05T06:40:55.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: ceph -s 2024-02-05T06:40:55.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: grep '4 up' 2024-02-05T06:40:56.233 INFO:tasks.workunit.client.0.smithi148.stdout: osd: 4 osds: 4 up (since 12s), 4 in (since 65s) 2024-02-05T06:40:56.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: ceph -s 2024-02-05T06:40:56.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: grep '4 in' 2024-02-05T06:40:56.621 INFO:tasks.workunit.client.0.smithi148.stdout: osd: 4 osds: 4 up (since 12s), 4 in (since 65s) 2024-02-05T06:40:56.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: ceph -s 2024-02-05T06:40:56.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: grep '1/1 objects unfound' 2024-02-05T06:40:56.975 INFO:tasks.workunit.client.0.smithi148.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:40:56.975 INFO:tasks.workunit.client.0.smithi148.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:40:56.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:40:56.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:40:56.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:40:56.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:40:56.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:40:56.977 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:40:56.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:40:56.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:40:56.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:40:57.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:40:57.291 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:40:57.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:40:57.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:40:57.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:40:57.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:40:57.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:40:57.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:40:57.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:40:57.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:40:57.297 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:40:57.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:40:57.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:40:57.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:40:57.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:40:57.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:40:57.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:40:57.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:40:57.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:40:57.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:40:57.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-05T06:40:57.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-repair 0 2024-02-05T06:40:57.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:40:57.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-05T06:40:57.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:40:57.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:40:57.329 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:40:57.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:40:57.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:40:57.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:40:57.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:40:57.332 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:40:57.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:40:57.334 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:40:57.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:40:57.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:40:57.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:40:57.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:40:57.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:40:57.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:40:57.337 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:40:57.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:40:57.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-05T06:40:57.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:40:57.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:40:57.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:40:57.342 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61355 2024-02-05T06:40:57.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61355 2024-02-05T06:40:57.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:40:57.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:40:57.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-05T06:40:57.344 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-05T06:40:57.344 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-05T06:40:57.356 INFO:tasks.workunit:Running workunit scrub/osd-scrub-snaps.sh... 2024-02-05T06:40:57.356 DEBUG:teuthology.orchestra.run.smithi148:workunit test scrub/osd-scrub-snaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh 2024-02-05T06:40:57.408 INFO:tasks.workunit.client.0.smithi148.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-05T06:40:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-05T06:40:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-05T06:40:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-05T06:40:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-05T06:40:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-05T06:40:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-05T06:40:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-05T06:40:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-05T06:40:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-05T06:40:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-snaps 2024-02-05T06:40:57.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:27: run: local dir=td/osd-scrub-snaps 2024-02-05T06:40:57.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:28: run: shift 2024-02-05T06:40:57.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: export CEPH_MON=127.0.0.1:7121 2024-02-05T06:40:57.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: CEPH_MON=127.0.0.1:7121 2024-02-05T06:40:57.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:31: run: export CEPH_ARGS 2024-02-05T06:40:57.415 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: uuidgen 2024-02-05T06:40:57.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: CEPH_ARGS+='--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none ' 2024-02-05T06:40:57.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:33: run: CEPH_ARGS+='--mon-host=127.0.0.1:7121 ' 2024-02-05T06:40:57.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:35: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-02-05T06:40:57.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: set 2024-02-05T06:40:57.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-05T06:40:57.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: local 'funcs=TEST_scrub_snaps 2024-02-05T06:40:57.419 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_scrub_snaps_primary 2024-02-05T06:40:57.419 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_scrub_snaps_replica' 2024-02-05T06:40:57.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-02-05T06:40:57.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-02-05T06:40:57.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-snaps 2024-02-05T06:40:57.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-snaps 2024-02-05T06:40:57.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-02-05T06:40:57.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:40:57.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-02-05T06:40:57.420 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:40:57.420 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:40:57.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:40:57.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:40:57.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:40:57.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:40:57.421 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:40:57.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:40:57.422 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:40:57.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:40:57.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:40:57.424 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:40:57.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:40:57.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:40:57.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:40:57.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:40:57.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:40:57.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:40:57.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-02-05T06:40:57.428 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:40:57.428 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:40:57.428 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:40:57.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.479281 2024-02-05T06:40:57.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:40:57.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:40:57.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-snaps 2024-02-05T06:40:57.431 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:40:57.431 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:40:57.431 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:40:57.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.479281 2024-02-05T06:40:57.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:40:57.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-05T06:40:57.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-05T06:40:57.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:40:57.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-02-05T06:40:57.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps td/osd-scrub-snaps 2024-02-05T06:40:57.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:171: TEST_scrub_snaps: local dir=td/osd-scrub-snaps 2024-02-05T06:40:57.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:172: TEST_scrub_snaps: local poolname=test 2024-02-05T06:40:57.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:173: TEST_scrub_snaps: local OBJS=16 2024-02-05T06:40:57.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:174: TEST_scrub_snaps: local OSDS=1 2024-02-05T06:40:57.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:176: TEST_scrub_snaps: TESTDATA=testdata.479281 2024-02-05T06:40:57.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:178: TEST_scrub_snaps: run_mon td/osd-scrub-snaps a --osd_pool_default_size=1 2024-02-05T06:40:57.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-snaps 2024-02-05T06:40:57.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:40:57.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:40:57.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:40:57.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-snaps/a 2024-02-05T06:40:57.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=1 2024-02-05T06:40:57.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:40:57.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:40:57.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:40:57.824 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:40:57.824 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:40:57.824 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:40:57.824 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:40:57.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 2024-02-05T06:40:58.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:40:58.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:40:58.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:40:58.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:40:58.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:40:58.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:40:58.027 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:40:58.028 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:40:58.028 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:40:58.028 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:40:58.028 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:40:58.028 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:40:58.028 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.479281/ceph-mon.a.asok 2024-02-05T06:40:58.029 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:40:58.029 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.479281/ceph-mon.a.asok config get fsid 2024-02-05T06:40:58.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:40:58.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:40:58.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:40:58.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:40:58.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:40:58.141 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:40:58.141 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:40:58.141 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:40:58.141 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:40:58.141 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:40:58.141 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:40:58.141 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.479281/ceph-mon.a.asok 2024-02-05T06:40:58.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:40:58.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.479281/ceph-mon.a.asok config get mon_host 2024-02-05T06:40:58.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:179: TEST_scrub_snaps: run_mgr td/osd-scrub-snaps x 2024-02-05T06:40:58.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-snaps 2024-02-05T06:40:58.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:40:58.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:40:58.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:40:58.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-snaps/x 2024-02-05T06:40:58.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:40:59.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:40:59.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:40:59.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:40:59.128 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:40:59.128 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:40:59.128 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:40:59.129 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:40:59.129 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:40:59.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:40:59.164 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: expr 1 - 1 2024-02-05T06:40:59.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: seq 0 0 2024-02-05T06:40:59.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:40:59.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:182: TEST_scrub_snaps: run_osd td/osd-scrub-snaps 0 2024-02-05T06:40:59.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:40:59.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:40:59.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:40:59.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:40:59.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-02-05T06:40:59.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-02-05T06:40:59.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:40:59.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:40:59.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:40:59.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-02-05T06:40:59.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:40:59.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:40:59.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:40:59.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:40:59.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:40:59.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:40:59.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:40:59.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:40:59.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-02-05T06:40:59.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:40:59.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ecd55bea-ca99-4ef4-b2ad-6f37e1b6ddbb 2024-02-05T06:40:59.172 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 ecd55bea-ca99-4ef4-b2ad-6f37e1b6ddbb 2024-02-05T06:40:59.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ecd55bea-ca99-4ef4-b2ad-6f37e1b6ddbb' 2024-02-05T06:40:59.172 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:40:59.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7gsBlESndChAAiCUmUtrJF4SONXXZ1Qn3zA== 2024-02-05T06:40:59.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7gsBlESndChAAiCUmUtrJF4SONXXZ1Qn3zA=="}' 2024-02-05T06:40:59.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ecd55bea-ca99-4ef4-b2ad-6f37e1b6ddbb -i td/osd-scrub-snaps/0/new.json 2024-02-05T06:40:59.480 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:40:59.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-02-05T06:40:59.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7gsBlESndChAAiCUmUtrJF4SONXXZ1Qn3zA== --osd-uuid ecd55bea-ca99-4ef4-b2ad-6f37e1b6ddbb 2024-02-05T06:40:59.528 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:40:59.520+0000 7f7fc38e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:40:59.547 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:40:59.540+0000 7f7fc38e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:40:59.549 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:40:59.544+0000 7f7fc38e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:40:59.550 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:40:59.544+0000 7f7fc38e1640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:40:59.550 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:40:59.544+0000 7f7fc38e1640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:40:59.550 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:40:59.544+0000 7f7fc38e1640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:40:59.648 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:40:59.640+0000 7f7fc38e1640 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-02-05T06:41:05.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-02-05T06:41:05.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:41:05.693 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:41:05.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:41:05.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:41:06.088 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:41:06.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:41:06.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:41:06.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:41:06.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:41:06.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:41:06.118 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:41:06.112+0000 7f0f7765c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:41:06.147 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:41:06.140+0000 7f0f7765c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:41:06.148 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:41:06.140+0000 7f0f7765c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:41:06.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:41:06.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:41:06.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:41:06.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:41:06.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:41:06.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:06.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:41:06.415 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:41:06.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:06.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:06.691 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:41:06.684+0000 7f0f7765c640 -1 Falling back to public interface 2024-02-05T06:41:06.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:07.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:07.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:07.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:41:07.703 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:41:07.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:07.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:08.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:08.982 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:41:08.976+0000 7f0f7765c640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:41:09.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:09.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:09.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:41:09.006 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:41:09.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:09.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:09.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:10.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:10.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:10.433 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:41:10.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:41:10.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:10.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:10.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:11.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:11.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:11.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:41:11.771 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:41:11.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:11.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:12.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:13.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:13.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:13.085 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:41:13.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:41:13.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:13.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:13.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:14.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:14.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:14.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:41:14.395 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:41:14.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:14.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:14.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:15.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:15.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:15.759 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:41:15.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:41:15.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:15.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:16.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:17.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:17.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:17.079 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:41:17.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:41:17.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:17.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:17.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:18.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:18.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:18.397 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:41:18.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:41:18.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:18.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:18.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:19.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:19.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:19.782 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:41:19.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:41:19.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:19.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:20.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:21.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:21.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:21.099 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:41:21.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:41:21.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:21.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:21.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:22.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:22.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:22.405 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:41:22.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:41:22.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:22.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:23.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:24.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:24.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:24.098 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:41:24.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:41:24.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:24.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:24.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:25.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:25.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:25.807 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:41:25.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:41:25.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:25.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:26.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:27.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:27.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:27.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:41:27.763 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:41:27.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:27.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:28.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:29.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:29.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:29.607 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:41:29.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:41:29.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:29.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:30.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:31.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:31.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:31.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:41:31.067 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:41:31.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:31.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:31.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:41:32.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:41:32.400 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:41:32.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:41:32.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:41:32.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:41:32.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:41:32.712 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/518759925,v1:127.0.0.1:6803/518759925] [v2:127.0.0.1:6804/518759925,v1:127.0.0.1:6805/518759925] exists,up ecd55bea-ca99-4ef4-b2ad-6f37e1b6ddbb 2024-02-05T06:41:32.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:41:32.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:41:32.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:41:32.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:186: TEST_scrub_snaps: ceph osd set noscrub 2024-02-05T06:41:33.131 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:41:33.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:187: TEST_scrub_snaps: ceph osd set nodeep-scrub 2024-02-05T06:41:33.666 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:41:33.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:190: TEST_scrub_snaps: create_pool test 1 1 2024-02-05T06:41:33.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-05T06:41:34.176 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'test' created 2024-02-05T06:41:34.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:41:35.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:191: TEST_scrub_snaps: wait_for_clean 2024-02-05T06:41:35.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:41:35.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:41:35.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:41:35.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:41:35.202 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:41:35.202 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:41:35.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:41:35.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:41:35.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:41:35.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:41:35.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:41:35.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:41:35.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:41:35.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:41:35.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:41:35.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:41:35.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:41:35.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:41:35.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:41:35.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836486 2024-02-05T06:41:35.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836486 2024-02-05T06:41:35.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486' 2024-02-05T06:41:35.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:41:35.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836486 2024-02-05T06:41:35.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:41:35.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:41:35.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836486 2024-02-05T06:41:35.803 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:41:35.804 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836486 2024-02-05T06:41:35.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836486 2024-02-05T06:41:35.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-02-05T06:41:35.804 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:41:36.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836486 2024-02-05T06:41:36.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:41:37.109 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:41:37.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:41:37.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836486 2024-02-05T06:41:37.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:41:38.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:41:38.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:41:38.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836486 2024-02-05T06:41:38.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:41:38.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:41:38.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:41:39.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:41:39.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:41:39.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:41:39.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:41:39.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:41:39.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:41:39.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:41:39.197 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:41:39.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:41:39.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:41:39.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:41:39.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:41:39.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:41:39.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:41:39.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:41:39.901 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: ceph osd dump 2024-02-05T06:41:39.901 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: grep '^pool.*['\'']test['\'']' 2024-02-05T06:41:39.901 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: awk '{ print $2 }' 2024-02-05T06:41:40.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: poolid=1 2024-02-05T06:41:40.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:194: TEST_scrub_snaps: dd if=/dev/urandom of=testdata.479281 bs=1032 count=1 2024-02-05T06:41:40.214 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records in 2024-02-05T06:41:40.214 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records out 2024-02-05T06:41:40.214 INFO:tasks.workunit.client.0.smithi148.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000112687 s, 9.2 MB/s 2024-02-05T06:41:40.214 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: seq 1 16 2024-02-05T06:41:40.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:40.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj1 testdata.479281 2024-02-05T06:41:40.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:40.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj2 testdata.479281 2024-02-05T06:41:40.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:40.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj3 testdata.479281 2024-02-05T06:41:40.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:40.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj4 testdata.479281 2024-02-05T06:41:40.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:40.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj5 testdata.479281 2024-02-05T06:41:40.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:40.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj6 testdata.479281 2024-02-05T06:41:40.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:40.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj7 testdata.479281 2024-02-05T06:41:40.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:40.893 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj8 testdata.479281 2024-02-05T06:41:40.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:40.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj9 testdata.479281 2024-02-05T06:41:41.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:41.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj10 testdata.479281 2024-02-05T06:41:41.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:41.169 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj11 testdata.479281 2024-02-05T06:41:41.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:41.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj12 testdata.479281 2024-02-05T06:41:41.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:41.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj13 testdata.479281 2024-02-05T06:41:41.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:41.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj14 testdata.479281 2024-02-05T06:41:41.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:41.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj15 testdata.479281 2024-02-05T06:41:41.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:41:41.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj16 testdata.479281 2024-02-05T06:41:41.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: get_primary test obj1 2024-02-05T06:41:41.714 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-02-05T06:41:41.715 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:41:41.715 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-02-05T06:41:41.715 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:41:42.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: local primary=0 2024-02-05T06:41:42.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:202: TEST_scrub_snaps: create_scenario td/osd-scrub-snaps test testdata.479281 0 2024-02-05T06:41:42.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-02-05T06:41:42.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-02-05T06:41:42.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.479281 2024-02-05T06:41:42.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2024-02-05T06:41:42.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-02-05T06:41:42.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-02-05T06:41:42.328 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap1 2024-02-05T06:41:42.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=1 2024-02-05T06:41:42.332 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records in 2024-02-05T06:41:42.332 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records out 2024-02-05T06:41:42.332 INFO:tasks.workunit.client.0.smithi148.stderr:256 bytes copied, 0.000190972 s, 1.3 MB/s 2024-02-05T06:41:42.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.479281 2024-02-05T06:41:42.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.479281 2024-02-05T06:41:42.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.479281 2024-02-05T06:41:42.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-02-05T06:41:42.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:41:42.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.479281 2024-02-05T06:41:42.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:41:42.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.479281 2024-02-05T06:41:42.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:41:42.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.479281 2024-02-05T06:41:43.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:41:43.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.479281 2024-02-05T06:41:43.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:41:43.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.479281 2024-02-05T06:41:43.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:41:43.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.479281 2024-02-05T06:41:43.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:41:43.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.479281 2024-02-05T06:41:43.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:41:43.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.479281 2024-02-05T06:41:43.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:41:43.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.479281 2024-02-05T06:41:43.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-02-05T06:41:43.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-02-05T06:41:43.787 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap2 2024-02-05T06:41:43.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=2 2024-02-05T06:41:43.791 INFO:tasks.workunit.client.0.smithi148.stderr:2+0 records in 2024-02-05T06:41:43.792 INFO:tasks.workunit.client.0.smithi148.stderr:2+0 records out 2024-02-05T06:41:43.792 INFO:tasks.workunit.client.0.smithi148.stderr:512 bytes copied, 0.000209897 s, 2.4 MB/s 2024-02-05T06:41:43.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.479281 2024-02-05T06:41:43.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-02-05T06:41:43.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-02-05T06:41:44.164 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap3 2024-02-05T06:41:44.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=3 2024-02-05T06:41:44.170 INFO:tasks.workunit.client.0.smithi148.stderr:3+0 records in 2024-02-05T06:41:44.170 INFO:tasks.workunit.client.0.smithi148.stderr:3+0 records out 2024-02-05T06:41:44.170 INFO:tasks.workunit.client.0.smithi148.stderr:768 bytes copied, 0.000164609 s, 4.7 MB/s 2024-02-05T06:41:44.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.479281 2024-02-05T06:41:44.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-02-05T06:41:44.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-02-05T06:41:44.499 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap4 2024-02-05T06:41:44.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=4 2024-02-05T06:41:44.505 INFO:tasks.workunit.client.0.smithi148.stderr:4+0 records in 2024-02-05T06:41:44.505 INFO:tasks.workunit.client.0.smithi148.stderr:4+0 records out 2024-02-05T06:41:44.505 INFO:tasks.workunit.client.0.smithi148.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000273574 s, 3.7 MB/s 2024-02-05T06:41:44.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.479281 2024-02-05T06:41:44.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.479281 2024-02-05T06:41:44.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-02-05T06:41:44.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-02-05T06:41:44.942 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap5 2024-02-05T06:41:44.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-02-05T06:41:44.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-02-05T06:41:45.148 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap6 2024-02-05T06:41:45.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=6 2024-02-05T06:41:45.153 INFO:tasks.workunit.client.0.smithi148.stderr:6+0 records in 2024-02-05T06:41:45.153 INFO:tasks.workunit.client.0.smithi148.stderr:6+0 records out 2024-02-05T06:41:45.153 INFO:tasks.workunit.client.0.smithi148.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.000259339 s, 5.9 MB/s 2024-02-05T06:41:45.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.479281 2024-02-05T06:41:45.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-02-05T06:41:45.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-02-05T06:41:45.470 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap7 2024-02-05T06:41:45.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-02-05T06:41:45.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-02-05T06:41:45.798 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-02-05T06:41:45.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-02-05T06:41:45.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:41:45.882 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:41:45.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:41:45.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:41:45.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:41:46.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:41:46.193 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2024-02-05T06:41:47.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:41:47.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-02-05T06:41:48.804 INFO:tasks.workunit.client.0.smithi148.stdout:WARNING: only removing head with clones present 2024-02-05T06:41:48.804 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:ff7b1f36:::obj1:head# 2024-02-05T06:41:49.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-02-05T06:41:49.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-02-05T06:41:50.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:41:50.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:41:51.032 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:c52c9666:::obj5:2# 2024-02-05T06:41:51.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-02-05T06:41:51.567 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-02-05T06:41:52.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:41:52.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:41:52.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:41:53.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:41:53.753 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:41:53.753 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:41:53.753 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:41:53.753 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:41:53.753 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:41:53.753 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:41:53.753 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:41:53.753 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:41:53.753 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:41:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:41:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:41:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:41:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:41:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:41:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:41:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:41:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:41:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:41:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:41:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:41:53.754 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:41:53.755 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:41:53.755 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:41:53.755 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:41:53.755 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:41:53.755 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:41:53.755 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:41:53.755 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:41:53.755 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:41:53.755 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:41:53.755 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:41:53.755 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:41:53.755 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:41:53.755 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:41:53.756 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:41:53.756 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:41:53.756 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:41:53.756 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:41:53.756 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:41:53.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:41:53.756 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:41:53.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:41:54.386 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:c52c9666:::obj5:1# 2024-02-05T06:41:54.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:41:56.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:41:56.249 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:41:56.249 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:41:56.249 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:41:56.249 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:41:56.249 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:41:56.249 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:41:56.249 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:41:56.250 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:41:56.250 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:41:56.250 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:41:56.250 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:41:56.250 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:41:56.250 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:41:56.250 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:41:56.250 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:41:56.250 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:41:56.250 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:41:56.250 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:41:56.250 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:41:56.250 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:41:56.251 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:41:56.251 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:41:56.251 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:41:56.251 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:41:56.251 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:41:56.251 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:41:56.251 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:41:56.251 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:41:56.251 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:41:56.251 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:41:56.252 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:41:56.252 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:41:56.252 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:41:56.252 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:41:56.252 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:41:56.252 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:41:56.252 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:41:56.252 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:41:56.252 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:41:56.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:41:56.253 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:41:56.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-02-05T06:41:56.256 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-02-05T06:41:56.256 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-02-05T06:41:57.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:41:57.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=18 2024-02-05T06:41:57.104 INFO:tasks.workunit.client.0.smithi148.stderr:18+0 records in 2024-02-05T06:41:57.104 INFO:tasks.workunit.client.0.smithi148.stderr:18+0 records out 2024-02-05T06:41:57.104 INFO:tasks.workunit.client.0.smithi148.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000237167 s, 19.4 MB/s 2024-02-05T06:41:57.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.479281 2024-02-05T06:41:58.276 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2024-02-05T06:41:59.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:41:59.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=15 2024-02-05T06:41:59.865 INFO:tasks.workunit.client.0.smithi148.stderr:15+0 records in 2024-02-05T06:41:59.866 INFO:tasks.workunit.client.0.smithi148.stderr:15+0 records out 2024-02-05T06:41:59.866 INFO:tasks.workunit.client.0.smithi148.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000256757 s, 15.0 MB/s 2024-02-05T06:41:59.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.479281 2024-02-05T06:42:01.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2024-02-05T06:42:01.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-02-05T06:42:02.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:02.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:42:03.077 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:0ee9ae15:::obj4:7# 2024-02-05T06:42:03.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:42:04.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:42:04.925 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:42:04.925 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:42:04.925 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:42:04.925 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:42:04.925 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:42:04.925 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:42:04.925 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:42:04.925 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:42:04.926 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:42:04.926 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:42:04.926 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:42:04.926 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:42:04.926 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:42:04.926 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:42:04.926 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:42:04.926 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:42:04.926 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:42:04.926 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:42:04.926 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:42:04.926 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:42:04.927 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:42:04.927 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:42:04.927 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:42:04.927 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:42:04.927 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:42:04.927 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:42:04.927 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:42:04.927 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:42:04.927 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:42:04.927 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:42:04.927 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:42:04.927 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:42:04.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:42:04.928 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:42:04.928 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:42:04.928 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:42:04.928 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:42:04.928 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:42:04.928 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:42:04.929 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:42:04.929 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2024-02-05T06:42:04.929 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-02-05T06:42:05.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:05.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:42:06.483 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:461f8b5e:::obj16:7# 2024-02-05T06:42:07.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:42:08.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:42:08.377 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:42:08.377 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:42:08.377 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:42:08.377 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:42:08.377 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:42:08.377 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:42:08.377 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:42:08.377 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:42:08.377 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:42:08.377 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:42:08.377 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:42:08.377 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:42:08.378 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:42:08.378 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:42:08.378 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:42:08.378 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:42:08.378 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:42:08.378 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:42:08.378 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:42:08.378 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:42:08.378 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:42:08.378 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:42:08.378 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:42:08.378 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:42:08.378 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:42:08.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:42:08.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-02-05T06:42:08.380 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2024-02-05T06:42:09.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:09.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-02-05T06:42:10.589 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:10.590 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-02-05T06:42:10.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:10.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=7 2024-02-05T06:42:10.593 INFO:tasks.workunit.client.0.smithi148.stderr:7+0 records in 2024-02-05T06:42:10.593 INFO:tasks.workunit.client.0.smithi148.stderr:7+0 records out 2024-02-05T06:42:10.593 INFO:tasks.workunit.client.0.smithi148.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 0.000187583 s, 9.6 MB/s 2024-02-05T06:42:10.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.479281 2024-02-05T06:42:12.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2024-02-05T06:42:12.768 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:42:13.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:13.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-02-05T06:42:14.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2024-02-05T06:42:15.180 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:42:15.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:15.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-02-05T06:42:17.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2024-02-05T06:42:17.985 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:42:18.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:18.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-02-05T06:42:19.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2024-02-05T06:42:20.405 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:42:21.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:21.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-02-05T06:42:22.388 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2024-02-05T06:42:23.069 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:42:23.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:23.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-02-05T06:42:24.501 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2024-02-05T06:42:25.180 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:42:25.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:25.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-02-05T06:42:26.884 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2024-02-05T06:42:27.698 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:42:27.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:27.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-02-05T06:42:29.204 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2024-02-05T06:42:29.937 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:42:30.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:30.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-02-05T06:42:31.577 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2024-02-05T06:42:32.315 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:42:33.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:33.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-02-05T06:42:34.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-02-05T06:42:34.354 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2024-02-05T06:42:35.046 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:42:35.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:42:35.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-02-05T06:42:36.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-02-05T06:42:36.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-02-05T06:42:36.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:204: TEST_scrub_snaps: rm -f testdata.479281 2024-02-05T06:42:36.690 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: expr 1 - 1 2024-02-05T06:42:36.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: seq 0 0 2024-02-05T06:42:36.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:42:36.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:208: TEST_scrub_snaps: activate_osd td/osd-scrub-snaps 0 2024-02-05T06:42:36.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:42:36.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:42:36.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:42:36.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:42:36.695 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:42:36.696 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:42:36.697 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:42:36.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:42:36.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:42:36.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:42:36.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:42:36.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:42:36.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:42:36.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:42:36.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:42:36.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:42:36.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:42:36.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-02-05T06:42:36.698 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:42:36.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:42:36.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:42:36.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-02-05T06:42:36.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:42:36.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:42:36.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:42:36.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:42:36.732 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:42:36.726+0000 7f10540cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:42:36.735 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:42:36.730+0000 7f10540cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:42:36.736 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:42:36.730+0000 7f10540cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:42:37.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:42:37.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:42:37.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:42:37.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:42:37.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:42:37.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:42:37.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:42:37.012 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:42:37.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:42:37.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:42:37.288 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:42:37.282+0000 7f10540cc640 -1 Falling back to public interface 2024-02-05T06:42:37.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:42:38.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:42:38.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:42:38.351 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:42:38.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:42:38.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:42:38.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:42:38.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:42:39.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:42:39.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:42:39.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:42:39.652 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:42:39.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:42:39.652 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:42:39.743 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:42:39.738+0000 7f10540cc640 -1 osd.0 18 log_to_monitors true 2024-02-05T06:42:39.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:42:40.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:42:40.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:42:40.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:42:40.966 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:42:40.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:42:40.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:42:41.276 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6802/1733256769,v1:127.0.0.1:6803/1733256769] [v2:127.0.0.1:6804/1733256769,v1:127.0.0.1:6805/1733256769] exists,up ecd55bea-ca99-4ef4-b2ad-6f37e1b6ddbb 2024-02-05T06:42:41.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:42:41.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:42:41.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:42:41.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:210: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 25 2024-02-05T06:42:41.418 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:42:41.418 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T06:42:41.418 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:42:41.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:211: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 5 2024-02-05T06:42:41.575 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:42:41.575 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:42:41.575 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:42:41.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:212: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max 1 2024-02-05T06:42:41.709 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:42:41.709 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_pg_stat_report_interval_max = '' (not observed, change may require restart) " 2024-02-05T06:42:41.709 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:42:41.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:215: TEST_scrub_snaps: wait_for_clean 2024-02-05T06:42:41.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:42:41.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:42:41.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:42:41.723 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:42:41.724 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:42:41.724 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:42:41.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:42:41.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:42:41.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:42:41.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:42:41.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:42:41.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:42:41.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:42:41.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:42:41.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:42:42.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:42:42.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:42:42.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:42:42.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:42:42.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313218 2024-02-05T06:42:42.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313218 2024-02-05T06:42:42.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-90194313218' 2024-02-05T06:42:42.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:42:42.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-90194313218 2024-02-05T06:42:42.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:42:42.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:42:42.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-90194313218 2024-02-05T06:42:42.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:42:42.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313218 2024-02-05T06:42:42.265 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 90194313218 2024-02-05T06:42:42.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 90194313218' 2024-02-05T06:42:42.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:42:42.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 90194313218 2024-02-05T06:42:42.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:42:43.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:42:43.581 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:42:43.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313218 -lt 90194313218 2024-02-05T06:42:43.919 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:42:43.920 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:42:43.920 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:42:44.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:42:44.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:42:44.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:42:44.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:42:44.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:42:44.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:42:44.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:42:44.357 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:42:44.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:42:44.638 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:42:44.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:42:44.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:42:45.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:42:45.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:42:45.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:42:45.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:217: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_max 2024-02-05T06:42:45.166 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:42:45.167 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_shallow_scrub_chunk_max": "25" 2024-02-05T06:42:45.167 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:42:45.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_min 2024-02-05T06:42:45.334 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:42:45.334 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_shallow_scrub_chunk_min": "5" 2024-02-05T06:42:45.334 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:42:45.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:219: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max 2024-02-05T06:42:45.502 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:42:45.561 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_pg_stat_report_interval_max": "1" 2024-02-05T06:42:45.561 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:42:45.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:220: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_max 2024-02-05T06:42:45.665 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:42:45.666 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_chunk_max": "25" 2024-02-05T06:42:45.666 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:42:45.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:221: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_min 2024-02-05T06:42:45.810 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:42:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_chunk_min": "5" 2024-02-05T06:42:45.811 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:42:45.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:223: TEST_scrub_snaps: local pgid=1.0 2024-02-05T06:42:45.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:224: TEST_scrub_snaps: pg_scrub 1.0 2024-02-05T06:42:45.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-02-05T06:42:45.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-02-05T06:42:45.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:42:45.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:42:45.826 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:42:45.826 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:42:46.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-02-05T06:41:34.050053+0000 2024-02-05T06:42:46.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-02-05T06:42:46.434 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 1.0 on osd.0 to scrub 2024-02-05T06:42:46.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-02-05T06:41:34.050053+0000 2024-02-05T06:42:46.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:42:46.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:41:34.050053+0000 2024-02-05T06:42:46.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:42:46.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:42:46.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:42:46.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:42:46.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:42:46.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:42:46.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:42:46.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:42:46.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:41:34.050053+0000 '>' 2024-02-05T06:41:34.050053+0000 2024-02-05T06:42:46.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:42:47.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:42:47.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:42:47.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:42:47.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:42:47.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:42:47.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:42:47.752 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:42:48.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:41:34.050053+0000 '>' 2024-02-05T06:41:34.050053+0000 2024-02-05T06:42:48.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:42:49.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:42:49.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:42:49.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:42:49.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:42:49.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:42:49.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:42:49.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:42:49.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:41:34.050053+0000 '>' 2024-02-05T06:41:34.050053+0000 2024-02-05T06:42:49.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:42:50.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:42:50.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:42:50.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:42:50.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:42:50.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:42:50.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:42:50.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:42:50.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:41:34.050053+0000 '>' 2024-02-05T06:41:34.050053+0000 2024-02-05T06:42:50.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:42:51.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:42:51.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:42:51.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:42:51.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:42:51.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:42:51.692 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:42:51.693 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:42:51.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:42:46.856480+0000 '>' 2024-02-05T06:41:34.050053+0000 2024-02-05T06:42:51.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:42:51.993 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:228: TEST_scrub_snaps: wc -l 2024-02-05T06:42:51.994 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:228: TEST_scrub_snaps: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:42:51.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:228: TEST_scrub_snaps: test 2 = 2 2024-02-05T06:42:51.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: rados list-inconsistent-pg test 2024-02-05T06:42:52.029 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:232: TEST_scrub_snaps: jq '. | length' td/osd-scrub-snaps/json 2024-02-05T06:42:52.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:232: TEST_scrub_snaps: test 1 = 1 2024-02-05T06:42:52.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: jq -r '.[0]' td/osd-scrub-snaps/json 2024-02-05T06:42:52.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: test 1.0 = 1.0 2024-02-05T06:42:52.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: rados list-inconsistent-obj 1.0 2024-02-05T06:42:52.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:241: TEST_scrub_snaps: jq .inconsistents 2024-02-05T06:42:52.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:241: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:42:52.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:248: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2024-02-05T06:42:52.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:248: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:42:52.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:249: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:42:52.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:42:52.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:251: TEST_scrub_snaps: rados list-inconsistent-snapset 1.0 2024-02-05T06:42:52.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:253: TEST_scrub_snaps: jq .inconsistents 2024-02-05T06:42:52.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:253: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:42:52.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:632: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2024-02-05T06:42:52.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:632: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:42:52.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:633: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:42:52.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:42:52.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: test no = yes 2024-02-05T06:42:52.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:639: TEST_scrub_snaps: test '' = yes 2024-02-05T06:42:52.269 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:644: TEST_scrub_snaps: find td/osd-scrub-snaps 2024-02-05T06:42:52.269 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:644: TEST_scrub_snaps: grep 'osd[^/]*\.pid' 2024-02-05T06:42:52.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:644: TEST_scrub_snaps: pidfiles=td/osd-scrub-snaps/osd.0.pid 2024-02-05T06:42:52.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: pids= 2024-02-05T06:42:52.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: for pidfile in ${pidfiles} 2024-02-05T06:42:52.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:648: TEST_scrub_snaps: cat td/osd-scrub-snaps/osd.0.pid 2024-02-05T06:42:52.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:648: TEST_scrub_snaps: pids+='498174 ' 2024-02-05T06:42:52.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:651: TEST_scrub_snaps: ERRORS=0 2024-02-05T06:42:52.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: seq 1 7 2024-02-05T06:42:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: for i in `seq 1 7` 2024-02-05T06:42:52.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: rados -p test rmsnap snap1 2024-02-05T06:42:52.476 INFO:tasks.workunit.client.0.smithi148.stdout:removed pool test snap snap1 2024-02-05T06:42:52.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: for i in `seq 1 7` 2024-02-05T06:42:52.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: rados -p test rmsnap snap2 2024-02-05T06:42:52.681 INFO:tasks.workunit.client.0.smithi148.stdout:removed pool test snap snap2 2024-02-05T06:42:52.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: for i in `seq 1 7` 2024-02-05T06:42:52.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: rados -p test rmsnap snap3 2024-02-05T06:42:52.883 INFO:tasks.workunit.client.0.smithi148.stdout:removed pool test snap snap3 2024-02-05T06:42:52.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: for i in `seq 1 7` 2024-02-05T06:42:52.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: rados -p test rmsnap snap4 2024-02-05T06:42:53.084 INFO:tasks.workunit.client.0.smithi148.stdout:removed pool test snap snap4 2024-02-05T06:42:53.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: for i in `seq 1 7` 2024-02-05T06:42:53.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: rados -p test rmsnap snap5 2024-02-05T06:42:53.285 INFO:tasks.workunit.client.0.smithi148.stdout:removed pool test snap snap5 2024-02-05T06:42:53.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: for i in `seq 1 7` 2024-02-05T06:42:53.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: rados -p test rmsnap snap6 2024-02-05T06:42:53.511 INFO:tasks.workunit.client.0.smithi148.stdout:removed pool test snap snap6 2024-02-05T06:42:53.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: for i in `seq 1 7` 2024-02-05T06:42:53.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: rados -p test rmsnap snap7 2024-02-05T06:42:53.745 INFO:tasks.workunit.client.0.smithi148.stdout:removed pool test snap snap7 2024-02-05T06:42:53.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: sleep 5 2024-02-05T06:42:58.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:658: TEST_scrub_snaps: local -i loop=0 2024-02-05T06:42:58.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:659: TEST_scrub_snaps: ceph pg dump pgs 2024-02-05T06:42:58.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:659: TEST_scrub_snaps: grep -q snaptrim 2024-02-05T06:42:59.044 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:42:59.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: grep -q snaptrim_error 2024-02-05T06:42:59.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: ceph pg dump pgs 2024-02-05T06:42:59.339 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:42:59.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: break 2024-02-05T06:42:59.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:672: TEST_scrub_snaps: ceph pg dump pgs 2024-02-05T06:42:59.642 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T06:42:59.642 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 35 0 0 0 0 23416 0 0 60 0 60 active+clean+inconsistent+snaptrim_error 2024-02-05T06:42:53.775214+0000 23'60 29:123 [0] 0 [0] 0 18'56 2024-02-05T06:42:46.856480+0000 0'0 2024-02-05T06:41:34.050053+0000 7 1 periodic scrub scheduled @ 2024-02-06T14:39:52.748740+0000 33 0 2024-02-05T06:42:59.642 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T06:42:59.642 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T06:42:59.643 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:42:59.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:674: TEST_scrub_snaps: for pid in $pids 2024-02-05T06:42:59.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:676: TEST_scrub_snaps: kill -0 498174 2024-02-05T06:42:59.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:683: TEST_scrub_snaps: kill_daemons td/osd-scrub-snaps 2024-02-05T06:42:59.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:42:59.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:42:59.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:42:59.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:42:59.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:43:04.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:43:04.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:685: TEST_scrub_snaps: declare -a err_strings 2024-02-05T06:43:04.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:686: TEST_scrub_snaps: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' 2024-02-05T06:43:04.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:687: TEST_scrub_snaps: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' 2024-02-05T06:43:04.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:688: TEST_scrub_snaps: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' 2024-02-05T06:43:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:689: TEST_scrub_snaps: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' 2024-02-05T06:43:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:690: TEST_scrub_snaps: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' 2024-02-05T06:43:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:691: TEST_scrub_snaps: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' 2024-02-05T06:43:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:692: TEST_scrub_snaps: err_strings[6]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' 2024-02-05T06:43:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:693: TEST_scrub_snaps: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' 2024-02-05T06:43:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:694: TEST_scrub_snaps: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' 2024-02-05T06:43:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:695: TEST_scrub_snaps: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' 2024-02-05T06:43:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:696: TEST_scrub_snaps: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' 2024-02-05T06:43:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:697: TEST_scrub_snaps: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' 2024-02-05T06:43:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:698: TEST_scrub_snaps: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' 2024-02-05T06:43:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:699: TEST_scrub_snaps: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' 2024-02-05T06:43:04.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' 2024-02-05T06:43:04.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:701: TEST_scrub_snaps: err_strings[15]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' 2024-02-05T06:43:04.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' 2024-02-05T06:43:04.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:703: TEST_scrub_snaps: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' 2024-02-05T06:43:04.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:704: TEST_scrub_snaps: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' 2024-02-05T06:43:04.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:705: TEST_scrub_snaps: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' 2024-02-05T06:43:04.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:706: TEST_scrub_snaps: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2024-02-05T06:43:04.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:707: TEST_scrub_snaps: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' 2024-02-05T06:43:04.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:708: TEST_scrub_snaps: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-02-05T06:43:04.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:04.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:04.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:04.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:04.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:04.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:04.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:04.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:04.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:04.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:04.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:04.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:04.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:04.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:04.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:04.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:43:05.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:43:05.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:719: TEST_scrub_snaps: '[' 0 '!=' 0 ']' 2024-02-05T06:43:05.046 INFO:tasks.workunit.client.0.smithi148.stdout:TEST PASSED 2024-02-05T06:43:05.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:725: TEST_scrub_snaps: echo 'TEST PASSED' 2024-02-05T06:43:05.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:726: TEST_scrub_snaps: return 0 2024-02-05T06:43:05.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-02-05T06:43:05.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-02-05T06:43:05.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:43:05.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-02-05T06:43:05.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:43:05.048 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:43:05.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:43:05.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:43:05.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:43:05.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:43:05.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:43:05.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:43:05.055 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:43:05.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:43:05.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:43:05.056 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:43:05.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:43:05.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:43:05.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:43:05.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:43:05.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:43:05.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:43:05.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-02-05T06:43:05.074 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:43:05.074 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:43:05.074 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:43:05.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.479281 2024-02-05T06:43:05.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:43:05.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:43:05.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-02-05T06:43:05.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-02-05T06:43:05.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-snaps 2024-02-05T06:43:05.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-snaps 2024-02-05T06:43:05.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-02-05T06:43:05.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:43:05.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-02-05T06:43:05.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:43:05.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:43:05.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:43:05.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:43:05.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:43:05.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:43:05.079 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:43:05.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:43:05.080 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:43:05.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:43:05.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:43:05.082 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:43:05.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:43:05.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:43:05.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:43:05.084 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:43:05.085 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:43:05.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:43:05.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-02-05T06:43:05.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:43:05.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:43:05.088 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:43:05.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.479281 2024-02-05T06:43:05.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:43:05.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:43:05.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-snaps 2024-02-05T06:43:05.091 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:43:05.091 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:43:05.091 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:43:05.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.479281 2024-02-05T06:43:05.092 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:43:05.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:43:05.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:43:05.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-02-05T06:43:05.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_primary td/osd-scrub-snaps 2024-02-05T06:43:05.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1174: TEST_scrub_snaps_primary: local dir=td/osd-scrub-snaps 2024-02-05T06:43:05.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1175: TEST_scrub_snaps_primary: ORIG_ARGS='--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1176: TEST_scrub_snaps_primary: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1177: TEST_scrub_snaps_primary: _scrub_snaps_multi td/osd-scrub-snaps primary 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:731: _scrub_snaps_multi: local poolname=test 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local OBJS=16 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local OSDS=2 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local which=primary 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: TESTDATA=testdata.479281 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-snaps 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-snaps/a 2024-02-05T06:43:05.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2024-02-05T06:43:05.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:43:05.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:43:05.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:43:05.495 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:43:05.495 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:43:05.495 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:43:05.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:43:05.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T06:43:05.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:43:05.707 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:43:05.707 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:43:05.707 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:43:05.707 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:43:05.708 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:43:05.708 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:43:05.708 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:43:05.709 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:43:05.709 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:43:05.709 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:43:05.709 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:43:05.709 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.479281/ceph-mon.a.asok 2024-02-05T06:43:05.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:43:05.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.479281/ceph-mon.a.asok config get fsid 2024-02-05T06:43:05.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:43:05.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:43:05.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:43:05.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:43:05.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:43:05.826 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:43:05.826 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:43:05.826 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:43:05.826 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:43:05.827 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:43:05.827 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:43:05.827 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.479281/ceph-mon.a.asok 2024-02-05T06:43:05.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:43:05.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.479281/ceph-mon.a.asok config get mon_host 2024-02-05T06:43:05.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:739: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2024-02-05T06:43:05.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-snaps 2024-02-05T06:43:05.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:43:05.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:43:05.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:43:05.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-snaps/x 2024-02-05T06:43:05.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:43:06.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:43:06.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:43:06.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:43:06.831 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:43:06.831 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:43:06.831 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:43:06.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:43:06.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:43:06.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:43:06.876 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: expr 2 - 1 2024-02-05T06:43:06.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: seq 0 1 2024-02-05T06:43:06.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:43:06.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2024-02-05T06:43:06.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:43:06.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:43:06.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:43:06.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:43:06.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-02-05T06:43:06.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:43:06.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:43:06.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:43:06.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:43:06.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:43:06.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:43:06.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:43:06.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:43:06.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:43:06.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:43:06.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:43:06.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:43:06.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:43:06.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-02-05T06:43:06.883 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:43:06.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fc3b78e7-b99e-4173-99e5-77ed2611d14c 2024-02-05T06:43:06.883 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 fc3b78e7-b99e-4173-99e5-77ed2611d14c 2024-02-05T06:43:06.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fc3b78e7-b99e-4173-99e5-77ed2611d14c' 2024-02-05T06:43:06.884 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:43:06.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB6g8BlCQe4NRAAxStUgoh1PpvySVoDY1u8Vw== 2024-02-05T06:43:06.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB6g8BlCQe4NRAAxStUgoh1PpvySVoDY1u8Vw=="}' 2024-02-05T06:43:06.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fc3b78e7-b99e-4173-99e5-77ed2611d14c -i td/osd-scrub-snaps/0/new.json 2024-02-05T06:43:07.184 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:43:07.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-02-05T06:43:07.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6g8BlCQe4NRAAxStUgoh1PpvySVoDY1u8Vw== --osd-uuid fc3b78e7-b99e-4173-99e5-77ed2611d14c 2024-02-05T06:43:07.247 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:07.241+0000 7fd9a13ee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:43:07.249 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:07.241+0000 7fd9a13ee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:43:07.250 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:07.245+0000 7fd9a13ee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:43:07.251 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:07.245+0000 7fd9a13ee640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:43:07.251 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:07.245+0000 7fd9a13ee640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:43:07.251 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:07.245+0000 7fd9a13ee640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:43:07.326 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:07.321+0000 7fd9a13ee640 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-02-05T06:43:13.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-02-05T06:43:13.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:43:13.407 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:43:13.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:43:13.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:43:13.809 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:43:13.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:43:13.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:43:13.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:43:13.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:43:13.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:43:13.854 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:13.845+0000 7f6137360640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:43:13.877 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:13.869+0000 7f6137360640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:43:13.878 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:13.873+0000 7f6137360640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:43:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:43:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:43:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:43:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:43:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:43:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:14.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:43:14.146 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:43:14.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:14.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:14.433 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:14.425+0000 7f6137360640 -1 Falling back to public interface 2024-02-05T06:43:14.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:15.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:15.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:15.464 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:43:15.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:43:15.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:15.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:15.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:16.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:16.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:16.794 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:43:16.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:43:16.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:16.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:16.943 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:16.937+0000 7f6137360640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:43:17.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:18.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:18.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:18.136 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:43:18.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:43:18.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:18.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:18.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:19.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:19.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:19.464 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:43:19.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:43:19.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:19.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:19.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:20.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:20.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:20.925 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:43:20.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:43:20.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:20.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:21.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:22.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:22.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:22.260 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:43:22.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:43:22.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:22.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:22.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:23.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:23.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:23.592 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:43:23.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:43:23.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:23.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:23.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:24.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:24.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:24.974 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:43:24.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:43:24.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:24.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:25.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:26.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:26.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:26.311 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:43:26.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:43:26.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:26.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:26.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:27.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:27.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:27.640 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:43:27.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:43:27.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:27.641 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:27.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:28.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:28.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:28.965 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:43:28.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:43:28.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:28.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:29.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:30.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:30.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:30.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:43:30.294 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:43:30.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:30.295 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:31.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:32.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:32.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:32.061 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:43:32.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:43:32.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:32.062 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:32.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:33.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:33.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:33.788 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:43:33.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:43:33.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:33.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:34.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:35.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:35.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:35.345 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:43:35.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:43:35.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:35.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:36.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:37.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:37.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:37.212 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:43:37.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:43:37.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:37.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:37.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:38.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:38.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:38.733 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:43:38.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:43:38.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:38.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:39.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:40.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:40.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:40.465 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:43:40.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:43:40.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:40.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:43:40.843 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/287362562,v1:127.0.0.1:6803/287362562] [v2:127.0.0.1:6804/287362562,v1:127.0.0.1:6805/287362562] exists,up fc3b78e7-b99e-4173-99e5-77ed2611d14c 2024-02-05T06:43:40.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:43:40.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:43:40.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:43:40.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:43:40.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2024-02-05T06:43:40.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:43:40.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:43:40.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:43:40.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:43:40.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/1 2024-02-05T06:43:40.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:43:40.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:43:40.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:43:40.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:43:40.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-02-05T06:43:40.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-02-05T06:43:40.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:43:40.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:43:40.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:43:40.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:43:40.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:43:40.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:43:40.849 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:43:40.849 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:43:40.849 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:43:40.849 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:43:40.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:43:40.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:43:40.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:43:40.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:43:40.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:43:40.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:43:40.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:43:40.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:43:40.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:43:40.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:43:40.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:43:40.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:43:40.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/1 2024-02-05T06:43:40.850 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:43:40.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6bc55dca-82d9-4224-a587-f2d10170fa16 2024-02-05T06:43:40.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6bc55dca-82d9-4224-a587-f2d10170fa16' 2024-02-05T06:43:40.851 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 6bc55dca-82d9-4224-a587-f2d10170fa16 2024-02-05T06:43:40.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:43:40.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCcg8BlFQD+MxAAR8Yict9b6BughQ/Y1mguwg== 2024-02-05T06:43:40.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCcg8BlFQD+MxAAR8Yict9b6BughQ/Y1mguwg=="}' 2024-02-05T06:43:40.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6bc55dca-82d9-4224-a587-f2d10170fa16 -i td/osd-scrub-snaps/1/new.json 2024-02-05T06:43:41.292 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:43:41.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/1/new.json 2024-02-05T06:43:41.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCcg8BlFQD+MxAAR8Yict9b6BughQ/Y1mguwg== --osd-uuid 6bc55dca-82d9-4224-a587-f2d10170fa16 2024-02-05T06:43:41.359 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:41.352+0000 7fdb7d923640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:43:41.384 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:41.376+0000 7fdb7d923640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:43:41.386 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:41.376+0000 7fdb7d923640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:43:41.387 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:41.380+0000 7fdb7d923640 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-02-05T06:43:41.387 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:41.380+0000 7fdb7d923640 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-02-05T06:43:41.387 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:41.380+0000 7fdb7d923640 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-02-05T06:43:41.456 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:41.448+0000 7fdb7d923640 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2024-02-05T06:43:47.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2024-02-05T06:43:47.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:43:47.797 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:43:47.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:43:47.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:43:48.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:43:48.237 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:43:48.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:43:48.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:43:48.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:43:48.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:43:48.279 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:48.272+0000 7fb5b893c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:43:48.299 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:48.292+0000 7fb5b893c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:43:48.300 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:48.292+0000 7fb5b893c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:43:48.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:43:48.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:43:48.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:43:48.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:43:48.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:43:48.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:48.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:43:48.552 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:43:48.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:43:48.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:48.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:48.869 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:48.860+0000 7fb5b893c640 -1 Falling back to public interface 2024-02-05T06:43:49.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:49.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:49.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:43:49.869 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:43:49.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:49.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:43:50.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:51.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:51.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:51.228 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:43:51.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:43:51.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:51.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:43:51.467 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:43:51.460+0000 7fb5b893c640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:43:51.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:52.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:52.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:52.601 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:43:52.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:43:52.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:52.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:43:52.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:53.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:53.935 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:43:53.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:53.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:43:53.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:53.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:43:54.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:55.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:55.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:55.270 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:43:55.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:43:55.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:55.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:43:55.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:56.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:56.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:56.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:43:56.713 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:43:56.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:56.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:43:57.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:58.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:58.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:58.039 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:43:58.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:43:58.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:58.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:43:58.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:43:59.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:43:59.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:43:59.419 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:43:59.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:43:59.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:43:59.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:43:59.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:44:00.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:44:00.751 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:44:00.751 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:44:00.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:44:00.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:44:00.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:44:01.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:44:02.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:44:02.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:44:02.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:44:02.171 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:44:02.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:44:02.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:44:02.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:44:03.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:44:03.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:44:03.620 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:44:03.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:44:03.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:44:03.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:44:03.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:44:04.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:44:04.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:44:04.945 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:44:04.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:44:04.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:44:04.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:44:06.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:44:07.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:44:07.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:44:07.058 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:44:07.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:44:07.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:44:07.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:44:07.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:44:08.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:44:08.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:44:08.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:44:08.623 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:44:08.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:44:08.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:44:09.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:44:10.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:44:10.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:44:10.186 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:44:10.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:44:10.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:44:10.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:44:10.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:44:11.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:44:11.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:44:11.582 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:44:11.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:44:11.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:44:11.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:44:11.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:44:12.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:44:12.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:44:12.972 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:44:12.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:44:12.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:44:12.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:44:13.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:44:14.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:44:14.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:44:14.448 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:44:14.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:44:14.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:44:14.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:44:14.945 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:44:15.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:44:15.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:44:15.947 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:44:15.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:44:15.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:44:15.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:44:16.282 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4103791456,v1:127.0.0.1:6811/4103791456] [v2:127.0.0.1:6812/4103791456,v1:127.0.0.1:6813/4103791456] exists,up 6bc55dca-82d9-4224-a587-f2d10170fa16 2024-02-05T06:44:16.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:44:16.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:44:16.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:44:16.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:746: _scrub_snaps_multi: ceph osd set noscrub 2024-02-05T06:44:16.902 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:44:16.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: ceph osd set nodeep-scrub 2024-02-05T06:44:17.455 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:44:17.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:750: _scrub_snaps_multi: create_pool test 1 1 2024-02-05T06:44:17.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-05T06:44:17.906 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'test' created 2024-02-05T06:44:17.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:44:18.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:751: _scrub_snaps_multi: wait_for_clean 2024-02-05T06:44:18.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:44:18.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:44:18.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:44:18.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:44:18.934 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:44:18.934 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:44:18.934 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:44:18.934 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:44:18.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:44:19.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:44:19.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:44:19.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:44:19.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:44:19.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:44:19.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:44:19.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:44:19.384 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:44:19.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:44:19.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:44:19.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:44:19.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-02-05T06:44:19.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-02-05T06:44:19.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-02-05T06:44:19.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:44:19.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:44:19.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-02-05T06:44:19.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-02-05T06:44:19.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672968' 2024-02-05T06:44:19.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:44:19.725 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:44:19.725 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-02-05T06:44:19.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:44:19.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-02-05T06:44:19.726 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:44:19.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-02-05T06:44:19.727 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836495 2024-02-05T06:44:19.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-02-05T06:44:19.728 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:44:20.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-02-05T06:44:20.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:44:21.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:44:21.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:44:21.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-02-05T06:44:21.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:44:22.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:44:22.386 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:44:22.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-02-05T06:44:22.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:44:22.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-02-05T06:44:22.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:44:22.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:44:22.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-02-05T06:44:22.720 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:44:22.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-02-05T06:44:22.721 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672968 2024-02-05T06:44:22.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-02-05T06:44:22.722 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:44:23.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672968 2024-02-05T06:44:23.057 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:44:23.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:44:23.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:44:23.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:44:23.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:44:23.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:44:23.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:44:23.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:44:23.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:44:23.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:44:23.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:44:23.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:44:23.785 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:44:23.785 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:44:23.786 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:44:24.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:44:24.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:44:24.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:44:24.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2024-02-05T06:44:24.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: awk '{ print $2 }' 2024-02-05T06:44:24.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: ceph osd dump 2024-02-05T06:44:24.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: poolid=1 2024-02-05T06:44:24.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.479281 bs=1032 count=1 2024-02-05T06:44:24.477 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records in 2024-02-05T06:44:24.477 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records out 2024-02-05T06:44:24.477 INFO:tasks.workunit.client.0.smithi148.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000115803 s, 8.9 MB/s 2024-02-05T06:44:24.478 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: seq 1 16 2024-02-05T06:44:24.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:24.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj1 testdata.479281 2024-02-05T06:44:24.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:24.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj2 testdata.479281 2024-02-05T06:44:24.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:24.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj3 testdata.479281 2024-02-05T06:44:24.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:24.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj4 testdata.479281 2024-02-05T06:44:25.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:25.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj5 testdata.479281 2024-02-05T06:44:25.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:25.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj6 testdata.479281 2024-02-05T06:44:25.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:25.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj7 testdata.479281 2024-02-05T06:44:25.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:25.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj8 testdata.479281 2024-02-05T06:44:25.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:25.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj9 testdata.479281 2024-02-05T06:44:25.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:25.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj10 testdata.479281 2024-02-05T06:44:26.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:26.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj11 testdata.479281 2024-02-05T06:44:26.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:26.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj12 testdata.479281 2024-02-05T06:44:26.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:26.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj13 testdata.479281 2024-02-05T06:44:26.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:26.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj14 testdata.479281 2024-02-05T06:44:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:26.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj15 testdata.479281 2024-02-05T06:44:26.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:44:26.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj16 testdata.479281 2024-02-05T06:44:27.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:760: _scrub_snaps_multi: get_primary test obj1 2024-02-05T06:44:27.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-02-05T06:44:27.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:44:27.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-02-05T06:44:27.077 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:44:27.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:760: _scrub_snaps_multi: local primary=1 2024-02-05T06:44:27.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:761: _scrub_snaps_multi: get_not_primary test obj1 2024-02-05T06:44:27.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-02-05T06:44:27.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-02-05T06:44:27.425 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-02-05T06:44:27.425 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-02-05T06:44:27.425 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:44:27.425 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-02-05T06:44:27.425 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:44:27.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T06:44:27.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T06:44:27.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-02-05T06:44:28.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:761: _scrub_snaps_multi: local replica=0 2024-02-05T06:44:28.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.479281 '$primary' 2024-02-05T06:44:28.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.479281 1 2024-02-05T06:44:28.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-02-05T06:44:28.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-02-05T06:44:28.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.479281 2024-02-05T06:44:28.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=1 2024-02-05T06:44:28.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-02-05T06:44:28.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-02-05T06:44:28.275 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap1 2024-02-05T06:44:28.278 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=1 2024-02-05T06:44:28.280 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records in 2024-02-05T06:44:28.280 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records out 2024-02-05T06:44:28.280 INFO:tasks.workunit.client.0.smithi148.stderr:256 bytes copied, 0.000244781 s, 1.0 MB/s 2024-02-05T06:44:28.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.479281 2024-02-05T06:44:28.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.479281 2024-02-05T06:44:28.707 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.479281 2024-02-05T06:44:28.866 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-02-05T06:44:28.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:44:28.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.479281 2024-02-05T06:44:29.024 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:44:29.025 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.479281 2024-02-05T06:44:29.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:44:29.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.479281 2024-02-05T06:44:29.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:44:29.334 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.479281 2024-02-05T06:44:29.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:44:29.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.479281 2024-02-05T06:44:29.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:44:29.652 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.479281 2024-02-05T06:44:29.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:44:29.802 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.479281 2024-02-05T06:44:29.944 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:44:29.945 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.479281 2024-02-05T06:44:30.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:44:30.112 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.479281 2024-02-05T06:44:30.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-02-05T06:44:30.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-02-05T06:44:30.464 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap2 2024-02-05T06:44:30.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=2 2024-02-05T06:44:30.469 INFO:tasks.workunit.client.0.smithi148.stderr:2+0 records in 2024-02-05T06:44:30.469 INFO:tasks.workunit.client.0.smithi148.stderr:2+0 records out 2024-02-05T06:44:30.469 INFO:tasks.workunit.client.0.smithi148.stderr:512 bytes copied, 0.000265941 s, 1.9 MB/s 2024-02-05T06:44:30.469 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.479281 2024-02-05T06:44:30.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-02-05T06:44:30.724 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-02-05T06:44:30.899 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap3 2024-02-05T06:44:30.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=3 2024-02-05T06:44:30.905 INFO:tasks.workunit.client.0.smithi148.stderr:3+0 records in 2024-02-05T06:44:30.905 INFO:tasks.workunit.client.0.smithi148.stderr:3+0 records out 2024-02-05T06:44:30.905 INFO:tasks.workunit.client.0.smithi148.stderr:768 bytes copied, 0.000267157 s, 2.9 MB/s 2024-02-05T06:44:30.905 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.479281 2024-02-05T06:44:31.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-02-05T06:44:31.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-02-05T06:44:31.447 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap4 2024-02-05T06:44:31.451 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=4 2024-02-05T06:44:31.452 INFO:tasks.workunit.client.0.smithi148.stderr:4+0 records in 2024-02-05T06:44:31.452 INFO:tasks.workunit.client.0.smithi148.stderr:4+0 records out 2024-02-05T06:44:31.452 INFO:tasks.workunit.client.0.smithi148.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00021164 s, 4.8 MB/s 2024-02-05T06:44:31.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.479281 2024-02-05T06:44:31.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.479281 2024-02-05T06:44:31.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-02-05T06:44:31.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-02-05T06:44:32.140 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap5 2024-02-05T06:44:32.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-02-05T06:44:32.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-02-05T06:44:32.345 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap6 2024-02-05T06:44:32.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=6 2024-02-05T06:44:32.351 INFO:tasks.workunit.client.0.smithi148.stderr:6+0 records in 2024-02-05T06:44:32.351 INFO:tasks.workunit.client.0.smithi148.stderr:6+0 records out 2024-02-05T06:44:32.351 INFO:tasks.workunit.client.0.smithi148.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.000194121 s, 7.9 MB/s 2024-02-05T06:44:32.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.479281 2024-02-05T06:44:32.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-02-05T06:44:32.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-02-05T06:44:32.793 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap7 2024-02-05T06:44:32.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-02-05T06:44:33.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-02-05T06:44:33.196 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-02-05T06:44:33.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-02-05T06:44:33.321 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:44:33.322 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:44:33.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:44:33.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:44:33.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:44:34.634 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:44:34.634 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj1 2024-02-05T06:44:36.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:44:36.441 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-02-05T06:44:37.244 INFO:tasks.workunit.client.0.smithi148.stdout:WARNING: only removing head with clones present 2024-02-05T06:44:37.245 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:ff7b1f36:::obj1:head# 2024-02-05T06:44:37.782 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-02-05T06:44:37.782 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-02-05T06:44:38.905 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:44:38.905 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:44:39.598 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:c52c9666:::obj5:2# 2024-02-05T06:44:40.153 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-02-05T06:44:40.154 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-02-05T06:44:41.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:44:41.446 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:44:41.446 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-02-05T06:44:42.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:44:42.530 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:44:42.531 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:44:42.531 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:42.531 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:44:42.531 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:44:42.531 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:44:42.531 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:44:42.531 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:44:42.531 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:44:42.531 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:44:42.531 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:42.531 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:44:42.531 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:44:42.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:44:42.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:44:42.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:44:42.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:44:42.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:42.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:42.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:44:42.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:44:42.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:42.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:44:42.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:42.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:44:42.532 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:42.533 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:44:42.533 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:42.533 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:44:42.533 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:42.533 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:44:42.533 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:44:42.533 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:44:42.533 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:42.533 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:44:42.533 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:44:42.533 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:44:42.533 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:42.533 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:44:42.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:44:42.534 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:44:42.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:44:43.137 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:c52c9666:::obj5:1# 2024-02-05T06:44:43.673 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-02-05T06:44:45.003 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:44:45.004 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:44:45.004 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:44:45.005 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:45.005 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:44:45.005 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:44:45.005 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:44:45.005 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:44:45.005 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:44:45.005 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:44:45.005 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:44:45.005 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:45.005 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:44:45.005 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:44:45.005 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:44:45.005 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:44:45.005 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:44:45.006 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:44:45.006 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:45.006 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:45.006 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:44:45.006 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:44:45.006 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:45.006 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:44:45.006 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:45.006 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:44:45.006 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:45.006 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:44:45.006 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:45.006 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:44:45.007 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:45.007 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:44:45.007 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:44:45.007 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:44:45.007 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:45.007 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:44:45.007 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:44:45.007 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:44:45.007 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:45.007 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:44:45.007 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:44:45.008 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:44:45.008 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-02-05T06:44:45.010 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-02-05T06:44:45.010 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-02-05T06:44:46.169 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:44:46.169 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=18 2024-02-05T06:44:46.170 INFO:tasks.workunit.client.0.smithi148.stderr:18+0 records in 2024-02-05T06:44:46.170 INFO:tasks.workunit.client.0.smithi148.stderr:18+0 records out 2024-02-05T06:44:46.170 INFO:tasks.workunit.client.0.smithi148.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.00022648 s, 20.3 MB/s 2024-02-05T06:44:46.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.479281 2024-02-05T06:44:47.311 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj3 2024-02-05T06:44:48.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:44:48.965 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=15 2024-02-05T06:44:48.967 INFO:tasks.workunit.client.0.smithi148.stderr:15+0 records in 2024-02-05T06:44:48.967 INFO:tasks.workunit.client.0.smithi148.stderr:15+0 records out 2024-02-05T06:44:48.967 INFO:tasks.workunit.client.0.smithi148.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.00024827 s, 15.5 MB/s 2024-02-05T06:44:48.967 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.479281 2024-02-05T06:44:50.210 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj4 2024-02-05T06:44:50.210 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-02-05T06:44:51.429 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:44:51.429 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:44:52.067 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:0ee9ae15:::obj4:7# 2024-02-05T06:44:52.601 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-02-05T06:44:53.681 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:44:53.682 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:44:53.682 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:53.682 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:44:53.682 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:44:53.682 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:44:53.682 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:44:53.682 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:44:53.682 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:44:53.683 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:44:53.683 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:53.683 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:44:53.683 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:44:53.683 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:44:53.683 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:44:53.683 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:44:53.683 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:53.683 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:53.683 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:44:53.683 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:53.683 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:44:53.683 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:53.683 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:53.684 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:44:53.684 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:53.684 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:53.684 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:44:53.684 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:44:53.684 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:53.684 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:44:53.684 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:44:53.684 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:53.684 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:44:53.685 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:44:53.685 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:53.685 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:53.685 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:53.685 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:53.685 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:53.685 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:53.685 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:44:53.686 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj16 2024-02-05T06:44:53.686 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-02-05T06:44:54.630 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:44:54.630 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:44:55.240 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:461f8b5e:::obj16:7# 2024-02-05T06:44:55.781 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-02-05T06:44:56.852 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:44:56.854 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:44:56.854 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:44:56.854 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:44:56.855 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:44:56.855 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:44:56.855 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:44:56.855 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:44:56.855 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:44:56.855 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:56.855 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:44:56.855 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:44:56.855 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:44:56.855 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:44:56.855 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:44:56.855 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:56.856 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:44:56.856 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:44:56.856 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:56.856 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:44:56.856 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:44:56.856 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:44:56.856 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:44:56.856 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:44:56.856 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:44:56.856 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:44:56.856 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:44:56.857 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-02-05T06:44:56.859 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj2 2024-02-05T06:44:57.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:44:57.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-02-05T06:44:58.874 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:44:58.878 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-02-05T06:44:58.879 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:44:58.879 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=7 2024-02-05T06:44:58.879 INFO:tasks.workunit.client.0.smithi148.stderr:7+0 records in 2024-02-05T06:44:58.879 INFO:tasks.workunit.client.0.smithi148.stderr:7+0 records out 2024-02-05T06:44:58.879 INFO:tasks.workunit.client.0.smithi148.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 0.000307133 s, 5.8 MB/s 2024-02-05T06:44:58.879 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.479281 2024-02-05T06:45:00.378 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj6 2024-02-05T06:45:01.117 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:45:01.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:45:01.670 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-02-05T06:45:02.898 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj7 2024-02-05T06:45:03.709 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:45:04.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:45:04.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-02-05T06:45:05.682 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj8 2024-02-05T06:45:06.494 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:45:07.028 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:45:07.028 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-02-05T06:45:08.182 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj9 2024-02-05T06:45:08.914 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:45:09.700 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:45:09.700 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-02-05T06:45:10.906 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj10 2024-02-05T06:45:11.644 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:45:12.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:45:12.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-02-05T06:45:13.298 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj11 2024-02-05T06:45:14.087 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:45:14.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:45:14.398 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-02-05T06:45:15.538 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj12 2024-02-05T06:45:16.417 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:45:16.953 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:45:16.953 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-02-05T06:45:18.154 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj13 2024-02-05T06:45:18.846 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:45:19.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:45:19.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-02-05T06:45:20.518 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj14 2024-02-05T06:45:21.282 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:45:22.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:45:22.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-02-05T06:45:23.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-02-05T06:45:23.290 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj15 2024-02-05T06:45:24.035 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:45:24.329 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:45:24.329 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-02-05T06:45:25.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-02-05T06:45:25.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-02-05T06:45:25.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: rm -f testdata.479281 2024-02-05T06:45:25.501 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: expr 2 - 1 2024-02-05T06:45:25.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: seq 0 1 2024-02-05T06:45:25.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:45:25.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2024-02-05T06:45:25.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:45:25.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:45:25.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:45:25.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:45:25.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-02-05T06:45:25.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:45:25.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:45:25.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:45:25.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:45:25.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-02-05T06:45:25.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-02-05T06:45:25.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:45:25.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:45:25.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:45:25.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:45:25.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:45:25.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:45:25.506 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:45:25.506 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:45:25.506 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:45:25.507 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:45:25.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:45:25.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:45:25.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:45:25.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:45:25.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:45:25.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:45:25.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:45:25.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:45:25.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:45:25.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:45:25.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-02-05T06:45:25.508 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:45:25.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:45:25.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:45:25.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-02-05T06:45:25.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:45:25.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:45:25.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:45:25.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:45:25.551 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:45:25.542+0000 7f684600e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:45:25.553 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:45:25.546+0000 7f684600e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:45:25.555 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:45:25.546+0000 7f684600e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:45:25.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:45:25.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:45:25.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:45:25.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:45:25.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:45:25.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:45:25.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:45:25.828 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:45:25.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:45:25.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:45:26.097 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:45:26.090+0000 7f684600e640 -1 Falling back to public interface 2024-02-05T06:45:26.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:45:27.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:45:27.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:45:27.159 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:45:27.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:45:27.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:45:27.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:45:27.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:45:28.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:45:28.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:45:28.487 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:45:28.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:45:28.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:45:28.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:45:28.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:45:29.222 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:45:29.214+0000 7f684600e640 -1 osd.0 23 log_to_monitors true 2024-02-05T06:45:29.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:45:29.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:45:29.819 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:45:29.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:45:29.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:45:29.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:45:30.178 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/3242260704,v1:127.0.0.1:6803/3242260704] [v2:127.0.0.1:6804/3242260704,v1:127.0.0.1:6805/3242260704] exists,up fc3b78e7-b99e-4173-99e5-77ed2611d14c 2024-02-05T06:45:30.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:45:30.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:45:30.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:45:30.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:45:30.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2024-02-05T06:45:30.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:45:30.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:45:30.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:45:30.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:45:30.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/1 2024-02-05T06:45:30.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:45:30.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:45:30.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:45:30.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:45:30.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-02-05T06:45:30.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-02-05T06:45:30.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:45:30.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:45:30.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:45:30.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:45:30.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:45:30.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:45:30.184 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:45:30.185 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:45:30.185 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:45:30.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:45:30.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:45:30.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:45:30.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:45:30.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:45:30.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:45:30.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:45:30.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:45:30.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:45:30.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:45:30.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:45:30.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/1 2024-02-05T06:45:30.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:45:30.186 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:45:30.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:45:30.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/1/whoami 2024-02-05T06:45:30.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:45:30.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:45:30.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:45:30.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:45:30.224 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:45:30.218+0000 7f489a183640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:45:30.226 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:45:30.218+0000 7f489a183640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:45:30.227 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:45:30.222+0000 7f489a183640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:45:30.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:45:30.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:45:30.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:45:30.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:45:30.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:45:30.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:45:30.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:45:30.493 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:45:30.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:45:30.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:45:30.761 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:45:30.754+0000 7f489a183640 -1 Falling back to public interface 2024-02-05T06:45:30.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:45:31.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:45:31.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:45:31.805 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:45:31.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:45:31.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:45:31.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:45:32.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:45:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:45:33.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:45:33.128 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:45:33.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:45:33.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:45:33.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:45:33.264 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:45:33.258+0000 7f489a183640 -1 osd.1 23 log_to_monitors true 2024-02-05T06:45:33.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:45:34.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:45:34.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:45:34.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:45:34.476 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:45:34.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:45:34.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:45:34.804 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/2758762301,v1:127.0.0.1:6811/2758762301] [v2:127.0.0.1:6812/2758762301,v1:127.0.0.1:6813/2758762301] exists,up 6bc55dca-82d9-4224-a587-f2d10170fa16 2024-02-05T06:45:34.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:45:34.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:45:34.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:45:34.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:772: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-02-05T06:45:34.954 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T06:45:34.955 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T06:45:34.955 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:45:34.968 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T06:45:34.968 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T06:45:34.968 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:45:34.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:773: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2024-02-05T06:45:35.124 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T06:45:35.124 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:45:35.124 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:45:35.137 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T06:45:35.137 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:45:35.137 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:45:35.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2024-02-05T06:45:35.286 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T06:45:35.286 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:45:35.286 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:45:35.299 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T06:45:35.299 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:45:35.299 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:45:35.313 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max 1 2024-02-05T06:45:35.490 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T06:45:35.490 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_pg_stat_report_interval_max = '' (not observed, change may require restart) " 2024-02-05T06:45:35.490 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:45:35.503 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T06:45:35.503 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_pg_stat_report_interval_max = '' (not observed, change may require restart) " 2024-02-05T06:45:35.503 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:45:35.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: wait_for_clean 2024-02-05T06:45:35.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:45:35.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:45:35.519 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:45:35.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:45:35.520 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:45:35.520 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:45:35.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:45:35.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:45:35.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:45:35.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:45:35.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:45:35.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:45:35.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:45:35.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:45:35.616 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:45:35.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:45:35.935 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:45:35.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:45:35.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:45:35.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:45:36.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149699 2024-02-05T06:45:36.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149699 2024-02-05T06:45:36.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149699' 2024-02-05T06:45:36.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:45:36.093 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:45:36.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-02-05T06:45:36.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-02-05T06:45:36.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149699 1-124554051586' 2024-02-05T06:45:36.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:45:36.250 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149699 2024-02-05T06:45:36.250 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:45:36.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:45:36.251 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149699 2024-02-05T06:45:36.251 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:45:36.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149699 2024-02-05T06:45:36.252 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 111669149699 2024-02-05T06:45:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2024-02-05T06:45:36.253 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:45:36.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149699 2024-02-05T06:45:36.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:45:37.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:45:37.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:45:37.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149699 -lt 111669149699 2024-02-05T06:45:37.872 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:45:37.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-124554051586 2024-02-05T06:45:37.873 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:45:37.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:45:37.875 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-124554051586 2024-02-05T06:45:37.876 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:45:37.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-02-05T06:45:37.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-02-05T06:45:37.878 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 124554051586 2024-02-05T06:45:37.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:45:38.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-02-05T06:45:38.195 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:45:38.196 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:45:38.196 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:45:38.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:45:38.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:45:38.587 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:45:38.587 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:45:38.588 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:45:38.588 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:45:38.588 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:45:38.588 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:45:38.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:45:38.920 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:45:38.921 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:45:38.921 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:45:39.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:45:39.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:45:39.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:45:39.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: local pgid=1.0 2024-02-05T06:45:39.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: pg_scrub 1.0 2024-02-05T06:45:39.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-02-05T06:45:39.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-02-05T06:45:39.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:45:39.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:45:39.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:45:39.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:45:39.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-02-05T06:44:17.842146+0000 2024-02-05T06:45:39.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-02-05T06:45:39.920 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 1.0 on osd.1 to scrub 2024-02-05T06:45:39.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-02-05T06:44:17.842146+0000 2024-02-05T06:45:39.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:45:39.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:44:17.842146+0000 2024-02-05T06:45:39.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:45:39.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:45:39.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:45:39.937 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:45:39.937 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:45:39.937 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:45:39.937 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:45:39.938 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:45:40.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:44:17.842146+0000 '>' 2024-02-05T06:44:17.842146+0000 2024-02-05T06:45:40.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:45:41.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:45:41.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:45:41.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:45:41.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:45:41.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:45:41.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:45:41.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:45:41.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:44:17.842146+0000 '>' 2024-02-05T06:44:17.842146+0000 2024-02-05T06:45:41.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:45:42.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:45:42.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:45:42.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:45:42.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:45:42.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:45:42.625 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:45:42.626 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:45:42.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:44:17.842146+0000 '>' 2024-02-05T06:44:17.842146+0000 2024-02-05T06:45:42.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:45:43.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:45:43.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:45:43.958 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:45:43.958 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:45:43.958 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:45:43.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:45:43.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:45:44.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:44:17.842146+0000 '>' 2024-02-05T06:44:17.842146+0000 2024-02-05T06:45:44.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:45:45.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:45:45.286 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:45:45.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:45:45.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:45:45.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:45:45.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:45:45.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:45:45.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:45:40.670734+0000 '>' 2024-02-05T06:44:17.842146+0000 2024-02-05T06:45:45.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:45:45.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:783: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:45:45.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:783: _scrub_snaps_multi: wc -l 2024-02-05T06:45:45.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:783: _scrub_snaps_multi: test 14 -gt 3 2024-02-05T06:45:45.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:784: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:45:45.620 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:784: _scrub_snaps_multi: wc -l 2024-02-05T06:45:45.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:784: _scrub_snaps_multi: test 14 -gt 3 2024-02-05T06:45:45.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: rados list-inconsistent-pg test 2024-02-05T06:45:45.663 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:788: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2024-02-05T06:45:45.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:788: _scrub_snaps_multi: test 1 = 1 2024-02-05T06:45:45.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:790: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2024-02-05T06:45:45.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:790: _scrub_snaps_multi: test 1.0 = 1.0 2024-02-05T06:45:45.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:792: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2024-02-05T06:45:45.750 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:45:45.750 INFO:tasks.workunit.client.0.smithi148.stdout: "epoch": 29, 2024-02-05T06:45:45.750 INFO:tasks.workunit.client.0.smithi148.stdout: "inconsistents": [ 2024-02-05T06:45:45.751 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.751 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.751 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj5", 2024-02-05T06:45:45.751 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.751 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.751 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:45:45.751 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 5 2024-02-05T06:45:45.751 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.751 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.751 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:45:45.751 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:45:45.751 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.751 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.751 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.752 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj5", 2024-02-05T06:45:45.752 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.752 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": 1, 2024-02-05T06:45:45.752 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1718170787, 2024-02-05T06:45:45.752 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.752 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.752 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.752 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.752 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'19", 2024-02-05T06:45:45.752 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'5", 2024-02-05T06:45:45.752 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4245.0:1", 2024-02-05T06:45:45.752 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 5, 2024-02-05T06:45:45.752 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.752 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:25.140360+0000", 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:25.141193+0000", 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x00799082", 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.753 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:45:45.754 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj5", 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 2, 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 20 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:45:45.755 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.756 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.756 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.756 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj5", 2024-02-05T06:45:45.756 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.756 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": 2, 2024-02-05T06:45:45.756 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1718170787, 2024-02-05T06:45:45.756 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.756 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.756 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.756 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.756 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "18'41", 2024-02-05T06:45:45.756 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "17'20", 2024-02-05T06:45:45.756 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4293.0:1", 2024-02-05T06:45:45.756 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 20, 2024-02-05T06:45:45.757 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.757 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:28.588216+0000", 2024-02-05T06:45:45.757 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:28.588987+0000", 2024-02-05T06:45:45.757 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.757 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.757 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.757 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.757 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.757 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.757 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.757 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0xabe731ae", 2024-02-05T06:45:45.757 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.757 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.757 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.758 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.758 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.758 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.758 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.758 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.758 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.758 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.758 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.758 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.758 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.758 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.758 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256 2024-02-05T06:45:45.758 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.759 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.759 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.759 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.759 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.759 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:45:45.759 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.759 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.759 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.759 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.759 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.759 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.759 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj5", 2024-02-05T06:45:45.759 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.759 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.760 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 4, 2024-02-05T06:45:45.760 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 42 2024-02-05T06:45:45.760 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.760 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.760 INFO:tasks.workunit.client.0.smithi148.stdout: "size_mismatch" 2024-02-05T06:45:45.760 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.760 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:45:45.760 INFO:tasks.workunit.client.0.smithi148.stdout: "size_mismatch_info", 2024-02-05T06:45:45.760 INFO:tasks.workunit.client.0.smithi148.stdout: "obj_size_info_mismatch" 2024-02-05T06:45:45.760 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.760 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.760 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.760 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj5", 2024-02-05T06:45:45.760 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": 4, 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1718170787, 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "20'45", 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "18'42", 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4329.0:1", 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 42, 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 512, 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:30.506798+0000", 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:30.513900+0000", 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.761 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.762 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.762 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.762 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.762 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.762 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.762 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x028aff3c", 2024-02-05T06:45:45.762 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.762 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.762 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.762 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.762 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.762 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.762 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.762 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 512 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: "size_mismatch_info", 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: "obj_size_info_mismatch" 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.763 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 4608, 2024-02-05T06:45:45.764 INFO:tasks.workunit.client.0.smithi148.stdout: "object_info": { 2024-02-05T06:45:45.764 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.764 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj5", 2024-02-05T06:45:45.764 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.764 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": 4, 2024-02-05T06:45:45.764 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1718170787, 2024-02-05T06:45:45.764 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.764 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.764 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.764 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.764 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "20'45", 2024-02-05T06:45:45.764 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "18'42", 2024-02-05T06:45:45.764 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4329.0:1", 2024-02-05T06:45:45.764 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 42, 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 512, 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:30.506798+0000", 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:30.513900+0000", 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x028aff3c", 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.765 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj4", 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 7, 2024-02-05T06:45:45.766 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 4 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj4", 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": 7, 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 2826278768, 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.767 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.768 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "23'51", 2024-02-05T06:45:45.768 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'4", 2024-02-05T06:45:45.768 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4242.0:1", 2024-02-05T06:45:45.768 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 4, 2024-02-05T06:45:45.768 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.768 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:24.988671+0000", 2024-02-05T06:45:45.768 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:24.989415+0000", 2024-02-05T06:45:45.768 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.768 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.768 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.768 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.768 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.768 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x00799082", 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.769 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.770 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.770 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032 2024-02-05T06:45:45.770 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.770 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.770 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.770 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.770 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.770 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:45:45.770 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.770 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.770 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.770 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.770 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.770 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.771 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj5", 2024-02-05T06:45:45.771 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.771 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.771 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 7, 2024-02-05T06:45:45.771 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 0 2024-02-05T06:45:45.771 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.771 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.771 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:45:45.771 INFO:tasks.workunit.client.0.smithi148.stdout: "missing", 2024-02-05T06:45:45.771 INFO:tasks.workunit.client.0.smithi148.stdout: "info_missing" 2024-02-05T06:45:45.771 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.771 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.771 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.772 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.772 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.772 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.772 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:45:45.772 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.772 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.772 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.772 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.772 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.772 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.772 INFO:tasks.workunit.client.0.smithi148.stdout: "info_missing" 2024-02-05T06:45:45.772 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.772 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1792 2024-02-05T06:45:45.772 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.773 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.773 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.773 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.773 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.773 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj1", 2024-02-05T06:45:45.773 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.773 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.773 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:45:45.773 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 18 2024-02-05T06:45:45.773 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.773 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.773 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:45:45.773 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:45:45.773 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj1", 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1828249343, 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'18", 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'1", 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4290.0:1", 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 18, 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.774 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:28.317928+0000", 2024-02-05T06:45:45.775 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:28.405196+0000", 2024-02-05T06:45:45.775 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.775 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.775 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.775 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.775 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.775 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.775 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.775 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0xabe731ae", 2024-02-05T06:45:45.775 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.775 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.775 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.775 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.775 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.776 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.776 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.776 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.776 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.776 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.776 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.776 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.776 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.776 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.776 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256 2024-02-05T06:45:45.776 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.776 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.776 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.776 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.777 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.777 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:45:45.777 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.777 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.777 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.777 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.777 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.777 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.777 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj10", 2024-02-05T06:45:45.777 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.777 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.777 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 32 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj10", 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 718195851, 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.778 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.779 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.779 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'32", 2024-02-05T06:45:45.779 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'10", 2024-02-05T06:45:45.779 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4311.0:1", 2024-02-05T06:45:45.779 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 32, 2024-02-05T06:45:45.779 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.779 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:29.539018+0000", 2024-02-05T06:45:45.779 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:29.539756+0000", 2024-02-05T06:45:45.779 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.779 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.779 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.779 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.779 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.779 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.780 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.780 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0xabe731ae", 2024-02-05T06:45:45.780 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.780 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.780 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.780 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.780 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.780 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.780 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.780 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.780 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.780 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.780 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.781 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.782 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.782 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.782 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.782 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.782 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.782 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.782 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.782 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.782 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.782 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:45:45.782 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:45:45.782 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.782 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:45:45.782 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.783 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "????", 2024-02-05T06:45:45.783 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:45:45.783 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:45:45.783 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.783 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.783 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.783 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.783 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.783 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.783 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.783 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.783 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.783 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj11", 2024-02-05T06:45:45.783 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 34 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj11", 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 693400951, 2024-02-05T06:45:45.784 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.785 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.785 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.785 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.785 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'34", 2024-02-05T06:45:45.785 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'11", 2024-02-05T06:45:45.785 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4314.0:1", 2024-02-05T06:45:45.785 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 34, 2024-02-05T06:45:45.785 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.785 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:29.690237+0000", 2024-02-05T06:45:45.785 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:29.690981+0000", 2024-02-05T06:45:45.785 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.785 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.785 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0xabe731ae", 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.786 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.787 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.787 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.787 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.787 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.787 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.787 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.787 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:45:45.787 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:45:45.787 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.787 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:45:45.787 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.787 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:45:45.787 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:45:45.787 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [] 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.788 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj13", 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 38 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:45:45.789 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj13", 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 2087409765, 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'38", 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'13", 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4320.0:1", 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 38, 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.790 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:29.983090+0000", 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:29.983846+0000", 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0xabe731ae", 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.791 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.792 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.792 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.792 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.792 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.792 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.792 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.792 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.792 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.792 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.792 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:45:45.792 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:45:45.792 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.792 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:45:45.792 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.793 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:45:45.793 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:45:45.793 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:45:45.793 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.793 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.793 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.793 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.793 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.793 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.793 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.793 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.793 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.793 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.793 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]" 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.794 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj14", 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 40 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj14", 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 2484217095, 2024-02-05T06:45:45.795 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.796 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.796 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.796 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.796 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'40", 2024-02-05T06:45:45.796 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'14", 2024-02-05T06:45:45.796 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4323.0:1", 2024-02-05T06:45:45.796 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 40, 2024-02-05T06:45:45.796 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.796 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:30.150098+0000", 2024-02-05T06:45:45.796 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:30.150888+0000", 2024-02-05T06:45:45.796 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.796 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.796 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.797 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.797 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.797 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.797 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.797 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0xabe731ae", 2024-02-05T06:45:45.797 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.797 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.797 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.797 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.797 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.797 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.797 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.797 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:45:45.798 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:45:45.799 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:45:45.799 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.799 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.799 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.799 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.799 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.799 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.799 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.799 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.799 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.799 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.799 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.799 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:45:45.799 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1033, 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.800 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj15", 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 15 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_corrupted" 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj15", 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 612772309, 2024-02-05T06:45:45.801 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "16'15", 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "0'0", 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4275.0:1", 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 15, 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:26.787637+0000", 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:26.788397+0000", 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.802 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x00799082", 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.803 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.804 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.804 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.804 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 0, 2024-02-05T06:45:45.804 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [] 2024-02-05T06:45:45.804 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.804 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.804 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.804 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.804 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.804 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.804 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_corrupted" 2024-02-05T06:45:45.804 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.804 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.804 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": "Z2FyYmFnZQo=" 2024-02-05T06:45:45.805 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.805 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.805 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.805 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.805 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.805 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj2", 2024-02-05T06:45:45.805 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.805 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.805 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:45:45.805 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 56 2024-02-05T06:45:45.805 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.805 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.805 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:45:45.805 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_missing" 2024-02-05T06:45:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj2", 2024-02-05T06:45:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:45:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1058988552, 2024-02-05T06:45:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "23'56", 2024-02-05T06:45:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "20'48", 2024-02-05T06:45:45.806 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4365.0:1", 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 56, 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 0, 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:33.234246+0000", 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:33.235039+0000", 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: "whiteout", 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty" 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0xffffffff", 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.807 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 0, 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 7, 2024-02-05T06:45:45.808 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 4, 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: 4, 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: 3, 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: 2, 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 7, 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1024, 2024-02-05T06:45:45.809 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: 7, 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: 6, 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: 5 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_missing" 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 0 2024-02-05T06:45:45.810 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj3", 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 44 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: "size_mismatch" 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:45:45.811 INFO:tasks.workunit.client.0.smithi148.stdout: "size_mismatch_info", 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: "obj_size_info_mismatch" 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj3", 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1643547569, 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "19'44", 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "17'22", 2024-02-05T06:45:45.812 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4335.0:1", 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 44, 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 768, 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:30.942935+0000", 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:30.991245+0000", 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x3d2a8b75", 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.813 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 768 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.814 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.815 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.815 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.815 INFO:tasks.workunit.client.0.smithi148.stdout: "size_mismatch_info", 2024-02-05T06:45:45.815 INFO:tasks.workunit.client.0.smithi148.stdout: "obj_size_info_mismatch" 2024-02-05T06:45:45.815 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.815 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 3840, 2024-02-05T06:45:45.815 INFO:tasks.workunit.client.0.smithi148.stdout: "object_info": { 2024-02-05T06:45:45.815 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.815 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj3", 2024-02-05T06:45:45.815 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.815 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:45:45.815 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1643547569, 2024-02-05T06:45:45.815 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.815 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.816 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.816 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.816 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "19'44", 2024-02-05T06:45:45.816 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "17'22", 2024-02-05T06:45:45.816 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4335.0:1", 2024-02-05T06:45:45.816 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 44, 2024-02-05T06:45:45.816 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 768, 2024-02-05T06:45:45.816 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:30.942935+0000", 2024-02-05T06:45:45.816 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:30.991245+0000", 2024-02-05T06:45:45.816 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.816 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.816 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.816 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.816 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x3d2a8b75", 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.817 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj6", 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 24 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.818 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj6", 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 2202164420, 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'24", 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'6", 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4299.0:1", 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 24, 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:28.905257+0000", 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:28.905969+0000", 2024-02-05T06:45:45.819 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.820 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.820 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.820 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.820 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.820 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.820 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.820 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0xabe731ae", 2024-02-05T06:45:45.820 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.820 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.820 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.820 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.820 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.820 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:45:45.821 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.822 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [] 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj7", 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 26 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.823 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj7", 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1552453721, 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'26", 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'7", 2024-02-05T06:45:45.824 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4302.0:1", 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 26, 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:29.064148+0000", 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:29.064973+0000", 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0xabe731ae", 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.825 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.826 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.826 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.826 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.826 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.826 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.826 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.826 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.826 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.826 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.826 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.826 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.826 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.826 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:45:45.826 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:45:45.827 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.827 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:45:45.827 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.827 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:45:45.827 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:45:45.827 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:45:45.827 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.827 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.827 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.827 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.827 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.827 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.827 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.828 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.828 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.828 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.828 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.828 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 0, 2024-02-05T06:45:45.828 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [] 2024-02-05T06:45:45.828 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.828 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.828 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.828 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.831 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.831 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.831 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj8", 2024-02-05T06:45:45.831 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.831 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.832 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:45:45.832 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 28 2024-02-05T06:45:45.832 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.832 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.832 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:45:45.832 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.832 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:45:45.832 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.832 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.832 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj8", 2024-02-05T06:45:45.832 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.832 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:45:45.832 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 2381834917, 2024-02-05T06:45:45.833 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.833 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.833 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.833 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.833 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'28", 2024-02-05T06:45:45.833 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'8", 2024-02-05T06:45:45.833 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4305.0:1", 2024-02-05T06:45:45.833 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 28, 2024-02-05T06:45:45.833 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.833 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:29.220497+0000", 2024-02-05T06:45:45.833 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:29.221224+0000", 2024-02-05T06:45:45.833 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.833 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.833 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.834 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.834 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.834 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.834 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.834 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0xabe731ae", 2024-02-05T06:45:45.834 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.834 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.834 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.834 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.834 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.834 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.834 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.834 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.835 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.835 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.835 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.835 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.835 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.835 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.835 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.835 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.835 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:45:45.835 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:45:45.835 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.835 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:45:45.835 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.835 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:45:45.836 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:45:45.836 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:45:45.836 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.836 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.836 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.836 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.836 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.836 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.836 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.836 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.836 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.836 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.836 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 0, 2024-02-05T06:45:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:45:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:45:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:45:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:45:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:45:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.837 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj9", 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 30 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:45:45.838 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj9", 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 3833113727, 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'30", 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'9", 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4308.0:1", 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 30, 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:44:29.393887+0000", 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:44:29.394694+0000", 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:45:45.839 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0xabe731ae", 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:45:45.840 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.841 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:45:45.841 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.841 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:45:45.841 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:45:45.841 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.841 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.841 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.841 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:45:45.841 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:45:45.841 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.841 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:45:45.841 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:45:45.841 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:45:45.841 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:45:45.842 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:45:45.842 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.842 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.842 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.842 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.842 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:45:45.842 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.842 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:45:45.842 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:45:45.842 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:45:45.842 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:45:45.842 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:45:45.842 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:45:45.842 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:45:45.843 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:45:45.843 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:45:45.843 INFO:tasks.workunit.client.0.smithi148.stdout: "size": "????", 2024-02-05T06:45:45.843 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:45:45.843 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:45:45.843 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:45:45.843 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.843 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.843 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.843 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.843 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.843 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.843 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:45:45.843 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:45:45.844 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:45:45.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:794: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2024-02-05T06:45:45.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: '[' primary = replica ']' 2024-02-05T06:45:45.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:808: _scrub_snaps_multi: scruberrors=30 2024-02-05T06:45:45.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:809: _scrub_snaps_multi: jq .inconsistents 2024-02-05T06:45:45.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:809: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:45:45.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1063: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2024-02-05T06:45:45.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1063: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:45:45.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1064: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:45:45.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:45:45.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1065: _scrub_snaps_multi: test no = yes 2024-02-05T06:45:45.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1070: _scrub_snaps_multi: test '' = yes 2024-02-05T06:45:45.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1075: _scrub_snaps_multi: find td/osd-scrub-snaps 2024-02-05T06:45:45.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1075: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2024-02-05T06:45:45.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1075: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2024-02-05T06:45:45.848 INFO:tasks.workunit.client.0.smithi148.stderr:td/osd-scrub-snaps/osd.0.pid' 2024-02-05T06:45:45.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1076: _scrub_snaps_multi: pids= 2024-02-05T06:45:45.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1077: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-02-05T06:45:45.849 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2024-02-05T06:45:45.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids+='520973 ' 2024-02-05T06:45:45.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1077: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-02-05T06:45:45.850 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2024-02-05T06:45:45.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids+='520409 ' 2024-02-05T06:45:45.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: ERRORS=0 2024-02-05T06:45:45.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1086: _scrub_snaps_multi: '[' primary = primary ']' 2024-02-05T06:45:45.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: seq 1 7 2024-02-05T06:45:45.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: for i in `seq 1 7` 2024-02-05T06:45:45.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1090: _scrub_snaps_multi: rados -p test rmsnap snap1 2024-02-05T06:45:46.052 INFO:tasks.workunit.client.0.smithi148.stdout:removed pool test snap snap1 2024-02-05T06:45:46.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: for i in `seq 1 7` 2024-02-05T06:45:46.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1090: _scrub_snaps_multi: rados -p test rmsnap snap2 2024-02-05T06:45:46.365 INFO:tasks.workunit.client.0.smithi148.stdout:removed pool test snap snap2 2024-02-05T06:45:46.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: for i in `seq 1 7` 2024-02-05T06:45:46.369 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1090: _scrub_snaps_multi: rados -p test rmsnap snap3 2024-02-05T06:45:46.554 INFO:tasks.workunit.client.0.smithi148.stdout:removed pool test snap snap3 2024-02-05T06:45:46.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: for i in `seq 1 7` 2024-02-05T06:45:46.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1090: _scrub_snaps_multi: rados -p test rmsnap snap4 2024-02-05T06:45:46.876 INFO:tasks.workunit.client.0.smithi148.stdout:removed pool test snap snap4 2024-02-05T06:45:46.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: for i in `seq 1 7` 2024-02-05T06:45:46.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1090: _scrub_snaps_multi: rados -p test rmsnap snap5 2024-02-05T06:45:47.066 INFO:tasks.workunit.client.0.smithi148.stdout:removed pool test snap snap5 2024-02-05T06:45:47.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: for i in `seq 1 7` 2024-02-05T06:45:47.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1090: _scrub_snaps_multi: rados -p test rmsnap snap6 2024-02-05T06:45:47.466 INFO:tasks.workunit.client.0.smithi148.stdout:removed pool test snap snap6 2024-02-05T06:45:47.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: for i in `seq 1 7` 2024-02-05T06:45:47.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1090: _scrub_snaps_multi: rados -p test rmsnap snap7 2024-02-05T06:45:47.709 INFO:tasks.workunit.client.0.smithi148.stdout:removed pool test snap snap7 2024-02-05T06:45:47.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1092: _scrub_snaps_multi: sleep 5 2024-02-05T06:45:52.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: local -i loop=0 2024-02-05T06:45:52.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1094: _scrub_snaps_multi: ceph pg dump pgs 2024-02-05T06:45:52.715 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1094: _scrub_snaps_multi: grep -q snaptrim 2024-02-05T06:45:53.024 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:45:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: ceph pg dump pgs 2024-02-05T06:45:53.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: grep -q snaptrim_error 2024-02-05T06:45:53.342 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:45:53.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1098: _scrub_snaps_multi: break 2024-02-05T06:45:53.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1108: _scrub_snaps_multi: ceph pg dump pgs 2024-02-05T06:45:53.646 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T06:45:53.646 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent+snaptrim_error 2024-02-05T06:45:47.738302+0000 23'56 37:119 [1,0] 1 [1,0] 1 23'56 2024-02-05T06:45:40.670734+0000 0'0 2024-02-05T06:44:17.842146+0000 7 1 periodic scrub scheduled @ 2024-02-06T09:08:30.189957+0000 33 0 2024-02-05T06:45:53.646 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T06:45:53.646 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T06:45:53.646 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:45:53.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1110: _scrub_snaps_multi: for pid in $pids 2024-02-05T06:45:53.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1112: _scrub_snaps_multi: kill -0 520973 2024-02-05T06:45:53.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1110: _scrub_snaps_multi: for pid in $pids 2024-02-05T06:45:53.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1112: _scrub_snaps_multi: kill -0 520409 2024-02-05T06:45:53.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2024-02-05T06:45:53.662 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:45:53.662 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:45:53.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:45:53.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:45:53.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:45:58.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:45:58.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1121: _scrub_snaps_multi: declare -a err_strings 2024-02-05T06:45:58.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2024-02-05T06:45:58.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1123: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2024-02-05T06:45:58.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2024-02-05T06:45:58.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2024-02-05T06:45:58.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2024-02-05T06:45:58.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2024-02-05T06:45:58.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2024-02-05T06:45:58.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' 2024-02-05T06:45:58.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:45:58.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:45:58.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:45:58.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:45:58.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:45:58.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:45:58.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:45:58.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:45:58.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:45:58.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:45:58.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:45:58.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:45:59.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:45:59.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:45:59.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:45:59.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:45:59.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1141: _scrub_snaps_multi: declare -a rep_err_strings 2024-02-05T06:45:59.008 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1142: _scrub_snaps_multi: eval echo '$primary' 2024-02-05T06:45:59.008 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1142: _scrub_snaps_multi: echo 1 2024-02-05T06:45:59.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1142: _scrub_snaps_multi: osd=1 2024-02-05T06:45:59.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1143: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-02-05T06:45:59.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2024-02-05T06:45:59.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:45:59.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1153: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2024-02-05T06:45:59.012 INFO:tasks.workunit.client.0.smithi148.stdout:TEST PASSED 2024-02-05T06:45:59.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1159: _scrub_snaps_multi: echo 'TEST PASSED' 2024-02-05T06:45:59.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1160: _scrub_snaps_multi: return 0 2024-02-05T06:45:59.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1178: TEST_scrub_snaps_primary: err=0 2024-02-05T06:45:59.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1179: TEST_scrub_snaps_primary: CEPH_ARGS='--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-02-05T06:45:59.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1180: TEST_scrub_snaps_primary: return 0 2024-02-05T06:45:59.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-02-05T06:45:59.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-02-05T06:45:59.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:45:59.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-02-05T06:45:59.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:45:59.014 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:45:59.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:45:59.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:45:59.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:45:59.021 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:45:59.022 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:45:59.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:45:59.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:45:59.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:45:59.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:45:59.024 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:45:59.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:45:59.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:45:59.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:45:59.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:45:59.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:45:59.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:45:59.028 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-02-05T06:45:59.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:45:59.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:45:59.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:45:59.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.479281 2024-02-05T06:45:59.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:45:59.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:45:59.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-02-05T06:45:59.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-02-05T06:45:59.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-snaps 2024-02-05T06:45:59.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-snaps 2024-02-05T06:45:59.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-02-05T06:45:59.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:45:59.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-02-05T06:45:59.048 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:45:59.048 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:45:59.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:45:59.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:45:59.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:45:59.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:45:59.050 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:45:59.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:45:59.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:45:59.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:45:59.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:45:59.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:45:59.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:45:59.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:45:59.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:45:59.055 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:45:59.056 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:45:59.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:45:59.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-02-05T06:45:59.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:45:59.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:45:59.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:45:59.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.479281 2024-02-05T06:45:59.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:45:59.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:45:59.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-snaps 2024-02-05T06:45:59.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:45:59.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:45:59.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:45:59.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.479281 2024-02-05T06:45:59.062 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:45:59.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:45:59.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:45:59.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-02-05T06:45:59.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_replica td/osd-scrub-snaps 2024-02-05T06:45:59.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1164: TEST_scrub_snaps_replica: local dir=td/osd-scrub-snaps 2024-02-05T06:45:59.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1165: TEST_scrub_snaps_replica: ORIG_ARGS='--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-02-05T06:45:59.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1166: TEST_scrub_snaps_replica: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:45:59.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1167: TEST_scrub_snaps_replica: _scrub_snaps_multi td/osd-scrub-snaps replica 2024-02-05T06:45:59.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2024-02-05T06:45:59.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:731: _scrub_snaps_multi: local poolname=test 2024-02-05T06:45:59.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local OBJS=16 2024-02-05T06:45:59.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local OSDS=2 2024-02-05T06:45:59.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local which=replica 2024-02-05T06:45:59.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: TESTDATA=testdata.479281 2024-02-05T06:45:59.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2024-02-05T06:45:59.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-snaps 2024-02-05T06:45:59.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:45:59.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:45:59.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:45:59.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-snaps/a 2024-02-05T06:45:59.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2024-02-05T06:45:59.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:45:59.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:45:59.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:45:59.458 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:45:59.458 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:45:59.459 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:45:59.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:45:59.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T06:45:59.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:45:59.667 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:45:59.667 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:45:59.667 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:45:59.667 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:45:59.668 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:45:59.668 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:45:59.669 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:45:59.669 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:45:59.669 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:45:59.669 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:45:59.669 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:45:59.669 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.479281/ceph-mon.a.asok 2024-02-05T06:45:59.669 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:45:59.669 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.479281/ceph-mon.a.asok config get fsid 2024-02-05T06:45:59.789 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:45:59.789 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:45:59.789 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:45:59.789 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:45:59.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:45:59.790 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:45:59.790 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:45:59.791 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:45:59.791 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:45:59.791 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:45:59.791 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:45:59.791 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.479281/ceph-mon.a.asok 2024-02-05T06:45:59.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:45:59.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.479281/ceph-mon.a.asok config get mon_host 2024-02-05T06:45:59.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:739: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2024-02-05T06:45:59.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-snaps 2024-02-05T06:45:59.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:45:59.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:45:59.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:45:59.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-snaps/x 2024-02-05T06:45:59.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:46:00.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:46:00.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:46:00.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:46:00.776 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:46:00.776 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:46:00.776 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:46:00.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:46:00.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:46:00.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:46:00.824 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: expr 2 - 1 2024-02-05T06:46:00.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: seq 0 1 2024-02-05T06:46:00.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:46:00.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2024-02-05T06:46:00.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:46:00.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:46:00.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:46:00.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:46:00.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-02-05T06:46:00.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:46:00.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:46:00.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:46:00.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:46:00.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-02-05T06:46:00.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-02-05T06:46:00.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:46:00.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:46:00.830 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:46:00.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:46:00.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:46:00.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:46:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:46:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:46:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:46:00.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:46:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:46:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:46:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:46:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:46:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:46:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:46:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:46:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:46:00.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:46:00.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:46:00.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:46:00.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:46:00.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-02-05T06:46:00.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:46:00.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b103e41-7e38-43d5-9ab7-87e6f63c4ec1 2024-02-05T06:46:00.833 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 4b103e41-7e38-43d5-9ab7-87e6f63c4ec1 2024-02-05T06:46:00.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4b103e41-7e38-43d5-9ab7-87e6f63c4ec1' 2024-02-05T06:46:00.833 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:46:00.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAohMBlNoWcMhAAPCj0/b5/97Sn3mqcI6Xziw== 2024-02-05T06:46:00.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAohMBlNoWcMhAAPCj0/b5/97Sn3mqcI6Xziw=="}' 2024-02-05T06:46:00.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b103e41-7e38-43d5-9ab7-87e6f63c4ec1 -i td/osd-scrub-snaps/0/new.json 2024-02-05T06:46:01.114 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:46:01.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-02-05T06:46:01.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAohMBlNoWcMhAAPCj0/b5/97Sn3mqcI6Xziw== --osd-uuid 4b103e41-7e38-43d5-9ab7-87e6f63c4ec1 2024-02-05T06:46:01.182 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:01.173+0000 7ff08d661640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:46:01.185 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:01.177+0000 7ff08d661640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:46:01.186 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:01.177+0000 7ff08d661640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:46:01.187 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:01.177+0000 7ff08d661640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:46:01.187 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:01.177+0000 7ff08d661640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:46:01.187 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:01.177+0000 7ff08d661640 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:46:01.274 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:01.265+0000 7ff08d661640 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-02-05T06:46:07.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-02-05T06:46:07.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:46:07.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:46:07.374 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:46:07.375 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:46:07.774 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:46:07.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:46:07.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:46:07.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:46:07.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:46:07.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:46:07.815 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:07.809+0000 7ff41abf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:46:07.840 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:07.833+0000 7ff41abf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:46:07.841 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:07.833+0000 7ff41abf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:46:08.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:46:08.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:46:08.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:46:08.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:46:08.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:46:08.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:08.114 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:46:08.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:46:08.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:08.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:08.414 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:08.405+0000 7ff41abf1640 -1 Falling back to public interface 2024-02-05T06:46:08.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:09.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:09.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:09.441 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:46:09.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:46:09.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:09.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:09.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:10.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:10.777 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:46:10.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:46:10.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:10.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:10.988 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:10.981+0000 7ff41abf1640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:46:11.110 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:12.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:12.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:12.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:46:12.112 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:46:12.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:12.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:12.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:13.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:13.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:13.443 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:46:13.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:46:13.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:13.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:13.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:14.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:14.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:14.791 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:46:14.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:46:14.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:14.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:15.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:16.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:16.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:16.117 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:46:16.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:46:16.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:16.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:16.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:17.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:17.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:17.437 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:46:17.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:46:17.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:17.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:17.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:18.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:18.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:18.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:46:18.785 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:46:18.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:18.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:19.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:20.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:20.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:20.117 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:46:20.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:46:20.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:20.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:20.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:21.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:21.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:21.447 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:46:21.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:46:21.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:21.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:21.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:22.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:22.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:22.793 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:46:22.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:46:22.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:22.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:23.119 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:24.120 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:24.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:24.121 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:46:24.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:46:24.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:24.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:24.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:25.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:25.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:25.777 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:46:25.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:46:25.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:25.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:26.415 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:27.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:27.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:27.417 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:46:27.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:46:27.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:27.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:28.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:29.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:29.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:29.574 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:46:29.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:46:29.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:29.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:30.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:31.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:31.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:31.435 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:46:31.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:46:31.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:31.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:31.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:32.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:32.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:32.859 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:46:32.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:46:32.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:32.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:33.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:34.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:34.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:34.423 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:46:34.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:46:34.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:34.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:34.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:35.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:35.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:35.920 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:46:35.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:46:35.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:35.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:46:36.248 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1528837754,v1:127.0.0.1:6803/1528837754] [v2:127.0.0.1:6804/1528837754,v1:127.0.0.1:6805/1528837754] exists,up 4b103e41-7e38-43d5-9ab7-87e6f63c4ec1 2024-02-05T06:46:36.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:46:36.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:46:36.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:46:36.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:46:36.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2024-02-05T06:46:36.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:46:36.252 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:46:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:46:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:46:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/1 2024-02-05T06:46:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:46:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:46:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:46:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:46:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-02-05T06:46:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-02-05T06:46:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:46:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:46:36.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:46:36.253 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:46:36.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:46:36.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:46:36.254 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:46:36.254 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:46:36.254 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:46:36.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:46:36.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:46:36.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:46:36.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:46:36.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:46:36.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:46:36.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:46:36.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:46:36.254 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:46:36.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:46:36.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:46:36.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:46:36.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:46:36.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/1 2024-02-05T06:46:36.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:46:36.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c3a55f10-c949-4e1d-a47b-325cc8c38ba9 2024-02-05T06:46:36.256 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 c3a55f10-c949-4e1d-a47b-325cc8c38ba9 2024-02-05T06:46:36.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c3a55f10-c949-4e1d-a47b-325cc8c38ba9' 2024-02-05T06:46:36.256 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:46:36.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBMhMBlT9F/EBAA958mmMpHrlGV058lFmrpUQ== 2024-02-05T06:46:36.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBMhMBlT9F/EBAA958mmMpHrlGV058lFmrpUQ=="}' 2024-02-05T06:46:36.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c3a55f10-c949-4e1d-a47b-325cc8c38ba9 -i td/osd-scrub-snaps/1/new.json 2024-02-05T06:46:36.669 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:46:36.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/1/new.json 2024-02-05T06:46:36.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMhMBlT9F/EBAA958mmMpHrlGV058lFmrpUQ== --osd-uuid c3a55f10-c949-4e1d-a47b-325cc8c38ba9 2024-02-05T06:46:36.736 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:36.728+0000 7f3df2ccb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:46:36.753 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:36.744+0000 7f3df2ccb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:46:36.754 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:36.744+0000 7f3df2ccb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:46:36.754 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:36.744+0000 7f3df2ccb640 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-02-05T06:46:36.755 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:36.744+0000 7f3df2ccb640 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-02-05T06:46:36.755 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:36.744+0000 7f3df2ccb640 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-02-05T06:46:36.832 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:36.824+0000 7f3df2ccb640 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2024-02-05T06:46:43.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2024-02-05T06:46:43.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:46:43.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:46:43.276 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:46:43.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:46:43.671 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:46:43.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:46:43.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:46:43.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:46:43.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:46:43.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:46:43.715 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:43.704+0000 7f8734daa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:46:43.733 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:43.724+0000 7f8734daa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:46:43.734 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:43.724+0000 7f8734daa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:46:44.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:46:44.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:46:44.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:46:44.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:46:44.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:46:44.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:44.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:46:44.026 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:46:44.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:44.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:46:44.282 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:44.272+0000 7f8734daa640 -1 Falling back to public interface 2024-02-05T06:46:44.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:45.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:45.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:45.350 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:46:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:46:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:45.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:46:45.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:46.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:46.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:46.668 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:46:46.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:46:46.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:46.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:46:46.931 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:46:46.920+0000 7f8734daa640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:46:46.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:47.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:47.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:48.000 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:46:48.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:46:48.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:48.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:46:48.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:49.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:49.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:49.476 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:46:49.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:46:49.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:49.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:46:49.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:50.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:50.804 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:46:50.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:50.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:46:50.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:50.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:46:51.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:52.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:52.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:52.222 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:46:52.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:46:52.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:52.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:46:52.622 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:53.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:53.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:53.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:46:53.624 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:46:53.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:53.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:46:53.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:55.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:55.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:55.000 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:46:55.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:46:55.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:55.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:46:55.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:56.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:56.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:56.339 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:46:56.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:46:56.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:56.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:46:56.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:57.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:57.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:57.808 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:46:57.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:46:57.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:57.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:46:58.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:46:59.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:46:59.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:46:59.224 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:46:59.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:46:59.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:46:59.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:46:59.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:47:00.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:47:00.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:47:00.555 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:47:00.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:47:00.555 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:47:00.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:47:01.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:47:02.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:47:02.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:47:02.125 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:47:02.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:47:02.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:47:02.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:47:02.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:47:04.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:47:04.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:47:04.001 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:47:04.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:47:04.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:47:04.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:47:04.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:47:05.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:47:05.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:47:05.811 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:47:05.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:47:05.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:47:05.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:47:06.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:47:07.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:47:07.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:47:07.223 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:47:07.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:47:07.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:47:07.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:47:07.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:47:08.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:47:08.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:47:08.814 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:47:08.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:47:08.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:47:08.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:47:09.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:47:10.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:47:10.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:47:10.309 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:47:10.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:47:10.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:47:10.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:47:11.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:47:12.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:47:12.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:47:12.054 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:47:12.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:47:12.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:47:12.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:47:12.410 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:47:13.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:47:13.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:47:13.412 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T06:47:13.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T06:47:13.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:47:13.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:47:13.732 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3191263801,v1:127.0.0.1:6811/3191263801] [v2:127.0.0.1:6812/3191263801,v1:127.0.0.1:6813/3191263801] exists,up c3a55f10-c949-4e1d-a47b-325cc8c38ba9 2024-02-05T06:47:13.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:47:13.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:47:13.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:47:13.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:746: _scrub_snaps_multi: ceph osd set noscrub 2024-02-05T06:47:14.285 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:47:14.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: ceph osd set nodeep-scrub 2024-02-05T06:47:14.820 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:47:14.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:750: _scrub_snaps_multi: create_pool test 1 1 2024-02-05T06:47:14.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-05T06:47:15.297 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'test' created 2024-02-05T06:47:15.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:47:16.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:751: _scrub_snaps_multi: wait_for_clean 2024-02-05T06:47:16.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:47:16.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:47:16.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:47:16.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:47:16.321 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:47:16.321 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:47:16.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:47:16.321 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:47:16.322 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:47:16.448 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:47:16.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:47:16.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:47:16.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:47:16.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:47:16.449 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:47:16.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:47:16.780 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:47:16.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:47:16.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:47:16.781 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:47:17.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-02-05T06:47:17.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-02-05T06:47:17.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-02-05T06:47:17.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:47:17.009 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:47:17.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-02-05T06:47:17.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-02-05T06:47:17.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672969' 2024-02-05T06:47:17.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:47:17.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:47:17.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-02-05T06:47:17.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:47:17.172 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-02-05T06:47:17.172 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:47:17.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-02-05T06:47:17.174 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836499 2024-02-05T06:47:17.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-02-05T06:47:17.174 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:47:17.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-02-05T06:47:17.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:47:17.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-02-05T06:47:17.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:47:17.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:47:17.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-02-05T06:47:17.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:47:17.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-02-05T06:47:17.497 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672969 2024-02-05T06:47:17.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-02-05T06:47:17.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:47:17.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672969 2024-02-05T06:47:17.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:47:17.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:47:17.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:47:18.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:47:18.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:47:18.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:47:18.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:47:18.192 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:47:18.193 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:47:18.193 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:47:18.193 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:47:18.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:47:18.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:47:18.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:47:18.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:47:18.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:47:18.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:47:18.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:47:18.862 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: ceph osd dump 2024-02-05T06:47:18.863 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2024-02-05T06:47:18.863 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: awk '{ print $2 }' 2024-02-05T06:47:19.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: poolid=1 2024-02-05T06:47:19.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.479281 bs=1032 count=1 2024-02-05T06:47:19.177 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records in 2024-02-05T06:47:19.178 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records out 2024-02-05T06:47:19.178 INFO:tasks.workunit.client.0.smithi148.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000113721 s, 9.1 MB/s 2024-02-05T06:47:19.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: seq 1 16 2024-02-05T06:47:19.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:19.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj1 testdata.479281 2024-02-05T06:47:19.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:19.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj2 testdata.479281 2024-02-05T06:47:19.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:19.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj3 testdata.479281 2024-02-05T06:47:19.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:19.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj4 testdata.479281 2024-02-05T06:47:19.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:19.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj5 testdata.479281 2024-02-05T06:47:19.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:19.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj6 testdata.479281 2024-02-05T06:47:20.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:20.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj7 testdata.479281 2024-02-05T06:47:20.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:20.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj8 testdata.479281 2024-02-05T06:47:20.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:20.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj9 testdata.479281 2024-02-05T06:47:20.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:20.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj10 testdata.479281 2024-02-05T06:47:20.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:20.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj11 testdata.479281 2024-02-05T06:47:20.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:20.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj12 testdata.479281 2024-02-05T06:47:21.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:21.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj13 testdata.479281 2024-02-05T06:47:21.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:21.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj14 testdata.479281 2024-02-05T06:47:21.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:21.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj15 testdata.479281 2024-02-05T06:47:21.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:47:21.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj16 testdata.479281 2024-02-05T06:47:21.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:760: _scrub_snaps_multi: get_primary test obj1 2024-02-05T06:47:21.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-02-05T06:47:21.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:47:21.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-02-05T06:47:21.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:47:21.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:760: _scrub_snaps_multi: local primary=1 2024-02-05T06:47:21.981 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:761: _scrub_snaps_multi: get_not_primary test obj1 2024-02-05T06:47:21.981 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-02-05T06:47:21.982 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-02-05T06:47:21.982 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-02-05T06:47:21.983 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-02-05T06:47:21.983 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:47:21.984 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-02-05T06:47:21.985 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:47:22.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T06:47:22.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-02-05T06:47:22.299 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T06:47:22.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:761: _scrub_snaps_multi: local replica=0 2024-02-05T06:47:22.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.479281 '$replica' 2024-02-05T06:47:22.621 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.479281 0 2024-02-05T06:47:22.621 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-02-05T06:47:22.621 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-02-05T06:47:22.621 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.479281 2024-02-05T06:47:22.621 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2024-02-05T06:47:22.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-02-05T06:47:22.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-02-05T06:47:22.824 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap1 2024-02-05T06:47:22.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=1 2024-02-05T06:47:22.829 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records in 2024-02-05T06:47:22.830 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records out 2024-02-05T06:47:22.830 INFO:tasks.workunit.client.0.smithi148.stderr:256 bytes copied, 0.000239871 s, 1.1 MB/s 2024-02-05T06:47:22.830 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.479281 2024-02-05T06:47:23.068 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.479281 2024-02-05T06:47:23.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.479281 2024-02-05T06:47:23.395 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-02-05T06:47:23.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:47:23.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.479281 2024-02-05T06:47:23.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:47:23.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.479281 2024-02-05T06:47:23.705 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:47:23.705 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.479281 2024-02-05T06:47:23.863 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:47:23.863 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.479281 2024-02-05T06:47:24.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:47:24.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.479281 2024-02-05T06:47:24.174 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:47:24.174 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.479281 2024-02-05T06:47:24.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:47:24.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.479281 2024-02-05T06:47:24.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:47:24.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.479281 2024-02-05T06:47:24.642 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:47:24.642 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.479281 2024-02-05T06:47:24.793 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-02-05T06:47:24.793 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-02-05T06:47:24.984 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap2 2024-02-05T06:47:24.989 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=2 2024-02-05T06:47:24.990 INFO:tasks.workunit.client.0.smithi148.stderr:2+0 records in 2024-02-05T06:47:24.990 INFO:tasks.workunit.client.0.smithi148.stderr:2+0 records out 2024-02-05T06:47:24.990 INFO:tasks.workunit.client.0.smithi148.stderr:512 bytes copied, 0.00016881 s, 3.0 MB/s 2024-02-05T06:47:24.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.479281 2024-02-05T06:47:25.270 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-02-05T06:47:25.270 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-02-05T06:47:25.504 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap3 2024-02-05T06:47:25.508 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=3 2024-02-05T06:47:25.509 INFO:tasks.workunit.client.0.smithi148.stderr:3+0 records in 2024-02-05T06:47:25.509 INFO:tasks.workunit.client.0.smithi148.stderr:3+0 records out 2024-02-05T06:47:25.509 INFO:tasks.workunit.client.0.smithi148.stderr:768 bytes copied, 0.00020082 s, 3.8 MB/s 2024-02-05T06:47:25.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.479281 2024-02-05T06:47:25.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-02-05T06:47:25.756 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-02-05T06:47:25.986 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap4 2024-02-05T06:47:25.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=4 2024-02-05T06:47:25.991 INFO:tasks.workunit.client.0.smithi148.stderr:4+0 records in 2024-02-05T06:47:25.991 INFO:tasks.workunit.client.0.smithi148.stderr:4+0 records out 2024-02-05T06:47:25.992 INFO:tasks.workunit.client.0.smithi148.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000200335 s, 5.1 MB/s 2024-02-05T06:47:25.992 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.479281 2024-02-05T06:47:26.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.479281 2024-02-05T06:47:26.417 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-02-05T06:47:26.417 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-02-05T06:47:26.592 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap5 2024-02-05T06:47:26.596 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-02-05T06:47:26.596 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-02-05T06:47:26.823 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap6 2024-02-05T06:47:26.826 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=6 2024-02-05T06:47:26.827 INFO:tasks.workunit.client.0.smithi148.stderr:6+0 records in 2024-02-05T06:47:26.827 INFO:tasks.workunit.client.0.smithi148.stderr:6+0 records out 2024-02-05T06:47:26.828 INFO:tasks.workunit.client.0.smithi148.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.000146443 s, 10.5 MB/s 2024-02-05T06:47:26.828 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.479281 2024-02-05T06:47:27.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-02-05T06:47:27.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-02-05T06:47:27.387 INFO:tasks.workunit.client.0.smithi148.stdout:created pool test snap snap7 2024-02-05T06:47:27.391 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-02-05T06:47:27.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-02-05T06:47:27.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-02-05T06:47:27.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-02-05T06:47:27.900 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:47:27.900 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:47:27.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:47:27.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:47:27.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:47:28.211 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:47:28.212 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2024-02-05T06:47:30.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:47:30.079 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-02-05T06:47:30.814 INFO:tasks.workunit.client.0.smithi148.stdout:WARNING: only removing head with clones present 2024-02-05T06:47:30.814 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:ff7b1f36:::obj1:head# 2024-02-05T06:47:31.351 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-02-05T06:47:31.352 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-02-05T06:47:32.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:47:32.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:47:33.109 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:c52c9666:::obj5:2# 2024-02-05T06:47:33.647 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-02-05T06:47:33.647 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-02-05T06:47:35.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:47:35.030 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:47:35.031 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:47:36.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:47:36.111 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:47:36.111 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:47:36.111 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:36.111 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:47:36.111 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:47:36.111 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:47:36.112 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:47:36.112 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:47:36.112 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:47:36.112 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:47:36.112 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:36.112 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:47:36.112 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:47:36.112 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:47:36.112 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:47:36.112 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:47:36.112 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:47:36.112 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:36.112 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:36.112 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:47:36.113 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:47:36.113 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:36.113 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:47:36.113 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:36.113 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:47:36.113 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:36.113 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:47:36.113 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:36.113 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:47:36.113 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:36.113 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:47:36.113 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:47:36.113 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:47:36.113 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:36.114 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:47:36.114 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:47:36.114 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:47:36.114 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:36.114 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:47:36.114 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:47:36.114 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:47:36.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:47:36.698 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:c52c9666:::obj5:1# 2024-02-05T06:47:37.234 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:47:38.557 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:47:38.559 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:47:38.559 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:47:38.559 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:38.559 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:47:38.559 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:47:38.559 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:47:38.560 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:47:38.560 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:47:38.560 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:47:38.560 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:47:38.560 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:38.560 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:47:38.560 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:47:38.560 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:47:38.560 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:47:38.560 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:47:38.560 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:47:38.560 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:38.560 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:38.560 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:47:38.561 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:47:38.561 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:38.561 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:47:38.561 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:38.561 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:47:38.561 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:38.561 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:47:38.561 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:38.561 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:47:38.561 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:38.561 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:47:38.561 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:47:38.561 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:47:38.561 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:38.562 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:47:38.562 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:47:38.562 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:47:38.562 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:38.562 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:47:38.562 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:47:38.562 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:47:38.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-02-05T06:47:38.563 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-02-05T06:47:38.563 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-02-05T06:47:39.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:47:39.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=18 2024-02-05T06:47:39.700 INFO:tasks.workunit.client.0.smithi148.stderr:18+0 records in 2024-02-05T06:47:39.700 INFO:tasks.workunit.client.0.smithi148.stderr:18+0 records out 2024-02-05T06:47:39.700 INFO:tasks.workunit.client.0.smithi148.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000243446 s, 18.9 MB/s 2024-02-05T06:47:39.700 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.479281 2024-02-05T06:47:40.846 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2024-02-05T06:47:42.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:47:42.430 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=15 2024-02-05T06:47:42.432 INFO:tasks.workunit.client.0.smithi148.stderr:15+0 records in 2024-02-05T06:47:42.432 INFO:tasks.workunit.client.0.smithi148.stderr:15+0 records out 2024-02-05T06:47:42.432 INFO:tasks.workunit.client.0.smithi148.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000328583 s, 11.7 MB/s 2024-02-05T06:47:42.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.479281 2024-02-05T06:47:43.626 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2024-02-05T06:47:43.626 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-02-05T06:47:44.706 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:47:44.707 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:47:45.295 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:0ee9ae15:::obj4:7# 2024-02-05T06:47:45.828 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:47:46.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:47:46.904 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:47:46.904 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:46.904 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:47:46.904 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:47:46.904 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:47:46.904 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:47:46.904 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:47:46.904 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:47:46.905 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:47:46.905 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:46.905 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:47:46.905 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:47:46.905 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:47:46.905 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:47:46.905 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:47:46.905 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:46.905 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:46.905 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:47:46.905 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:46.906 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:47:46.906 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:46.906 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:46.906 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:47:46.906 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:46.906 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:46.906 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:47:46.906 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:47:46.906 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:46.906 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:47:46.906 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:47:46.907 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:46.907 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:47:46.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:47:46.907 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:46.907 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:46.907 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:46.907 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:46.908 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:46.908 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:46.908 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:47:46.908 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2024-02-05T06:47:46.908 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-02-05T06:47:48.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:47:48.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:47:48.635 INFO:tasks.workunit.client.0.smithi148.stdout:remove #1:461f8b5e:::obj16:7# 2024-02-05T06:47:49.169 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:47:50.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:47:50.500 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:47:50.500 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:47:50.500 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:47:50.500 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:47:50.500 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:47:50.500 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:47:50.500 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:47:50.501 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:47:50.501 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:50.501 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:47:50.501 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:47:50.501 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:47:50.501 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:47:50.501 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:47:50.501 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:50.501 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:47:50.501 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:47:50.501 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:50.501 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:47:50.501 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:47:50.501 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:47:50.502 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:47:50.502 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:47:50.502 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:47:50.502 INFO:tasks.workunit.client.0.smithi148.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:47:50.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:47:50.502 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-02-05T06:47:50.505 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2024-02-05T06:47:51.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:47:51.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-02-05T06:47:52.827 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-02-05T06:47:52.827 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:47:52.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:47:52.829 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.479281 bs=256 count=7 2024-02-05T06:47:52.831 INFO:tasks.workunit.client.0.smithi148.stderr:7+0 records in 2024-02-05T06:47:52.831 INFO:tasks.workunit.client.0.smithi148.stderr:7+0 records out 2024-02-05T06:47:52.831 INFO:tasks.workunit.client.0.smithi148.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 0.00030599 s, 5.9 MB/s 2024-02-05T06:47:52.831 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.479281 2024-02-05T06:47:54.239 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2024-02-05T06:47:55.054 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:47:55.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:47:55.350 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-02-05T06:47:56.523 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2024-02-05T06:47:57.331 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:47:58.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:47:58.119 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-02-05T06:47:59.247 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2024-02-05T06:48:00.087 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:48:00.383 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:48:00.383 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-02-05T06:48:01.527 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2024-02-05T06:48:02.341 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:48:03.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:48:03.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-02-05T06:48:04.237 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2024-02-05T06:48:05.004 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:48:05.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:48:05.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-02-05T06:48:06.814 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2024-02-05T06:48:07.602 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:48:08.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:48:08.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-02-05T06:48:09.295 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2024-02-05T06:48:10.205 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:48:10.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:48:10.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-02-05T06:48:11.623 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2024-02-05T06:48:12.416 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:48:12.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:48:12.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-02-05T06:48:14.135 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2024-02-05T06:48:14.868 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:48:15.655 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:48:15.655 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-02-05T06:48:16.849 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-02-05T06:48:16.849 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2024-02-05T06:48:17.616 INFO:tasks.workunit.client.0.smithi148.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:48:18.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:48:18.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-02-05T06:48:19.254 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-02-05T06:48:19.255 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-02-05T06:48:19.255 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: rm -f testdata.479281 2024-02-05T06:48:19.258 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: expr 2 - 1 2024-02-05T06:48:19.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: seq 0 1 2024-02-05T06:48:19.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:48:19.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2024-02-05T06:48:19.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:48:19.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:48:19.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:48:19.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:48:19.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-02-05T06:48:19.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:48:19.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:48:19.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:48:19.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:48:19.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-02-05T06:48:19.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-02-05T06:48:19.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:48:19.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:48:19.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:48:19.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:48:19.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:48:19.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:48:19.265 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:48:19.265 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:48:19.265 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:48:19.265 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:48:19.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:48:19.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:48:19.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:48:19.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:48:19.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:48:19.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:48:19.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:48:19.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:48:19.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:48:19.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:48:19.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-02-05T06:48:19.266 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:48:19.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:48:19.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:48:19.268 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-02-05T06:48:19.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:48:19.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:48:19.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:48:19.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:48:19.309 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:19.302+0000 7fd136d2d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:19.311 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:19.302+0000 7fd136d2d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:19.313 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:19.306+0000 7fd136d2d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:19.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:48:19.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:48:19.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:48:19.590 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:48:19.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:48:19.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:48:19.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:48:19.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:48:19.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:48:19.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:48:19.862 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:19.854+0000 7fd136d2d640 -1 Falling back to public interface 2024-02-05T06:48:19.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:48:20.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:48:20.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:48:20.953 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:48:20.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:48:20.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:48:20.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:48:21.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:48:22.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:48:22.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:48:22.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:48:22.275 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:48:22.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:48:22.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:48:22.489 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:22.482+0000 7fd136d2d640 -1 osd.0 23 log_to_monitors true 2024-02-05T06:48:22.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:48:23.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:48:23.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:48:23.612 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:48:23.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:48:23.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:48:23.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:48:23.952 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/739097057,v1:127.0.0.1:6803/739097057] [v2:127.0.0.1:6804/739097057,v1:127.0.0.1:6805/739097057] exists,up 4b103e41-7e38-43d5-9ab7-87e6f63c4ec1 2024-02-05T06:48:23.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:48:23.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:48:23.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:48:23.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:48:23.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2024-02-05T06:48:23.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:48:23.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:48:23.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:48:23.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:48:23.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/1 2024-02-05T06:48:23.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:48:23.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:48:23.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:48:23.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:48:23.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-02-05T06:48:23.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-02-05T06:48:23.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:48:23.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:48:23.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:48:23.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:48:23.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:48:23.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:48:23.956 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:48:23.957 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:48:23.957 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:48:23.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:48:23.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' 2024-02-05T06:48:23.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:48:23.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:48:23.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:48:23.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:48:23.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:48:23.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:48:23.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:48:23.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:48:23.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:48:23.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/1 2024-02-05T06:48:23.958 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:48:23.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:48:23.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.479281/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:48:23.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/1/whoami 2024-02-05T06:48:23.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:48:23.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:48:23.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:48:23.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:48:23.997 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:23.990+0000 7fa2f0bc7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:24.011 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:24.002+0000 7fa2f0bc7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:24.012 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:24.006+0000 7fa2f0bc7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:24.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:48:24.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:48:24.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:48:24.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:48:24.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:48:24.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:48:24.318 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:48:24.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:48:24.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:48:24.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:48:24.555 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:24.546+0000 7fa2f0bc7640 -1 Falling back to public interface 2024-02-05T06:48:24.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:48:25.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:48:25.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:48:25.632 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:48:25.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:48:25.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:48:25.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:48:25.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:48:26.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:48:26.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:48:26.961 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:48:26.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:48:26.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:48:26.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:48:27.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:48:27.786 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:27.778+0000 7fa2f0bc7640 -1 osd.1 23 log_to_monitors true 2024-02-05T06:48:28.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:48:28.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:48:28.353 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:48:28.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:48:28.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:48:28.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:48:28.736 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/930561290,v1:127.0.0.1:6811/930561290] [v2:127.0.0.1:6812/930561290,v1:127.0.0.1:6813/930561290] exists,up c3a55f10-c949-4e1d-a47b-325cc8c38ba9 2024-02-05T06:48:28.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:48:28.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:48:28.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:48:28.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:772: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-02-05T06:48:28.887 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T06:48:28.887 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T06:48:28.887 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:48:28.900 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T06:48:28.900 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T06:48:28.900 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:48:28.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:773: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2024-02-05T06:48:29.059 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T06:48:29.059 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:48:29.059 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:48:29.073 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T06:48:29.073 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:48:29.073 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:48:29.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2024-02-05T06:48:29.281 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T06:48:29.281 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:48:29.281 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:48:29.295 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T06:48:29.295 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:48:29.295 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:48:29.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max 1 2024-02-05T06:48:29.462 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T06:48:29.463 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_pg_stat_report_interval_max = '' (not observed, change may require restart) " 2024-02-05T06:48:29.463 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:48:29.475 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T06:48:29.475 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_pg_stat_report_interval_max = '' (not observed, change may require restart) " 2024-02-05T06:48:29.475 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:48:29.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: wait_for_clean 2024-02-05T06:48:29.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:48:29.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:48:29.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:48:29.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:48:29.493 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:48:29.493 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:48:29.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:48:29.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:48:29.493 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:48:29.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:48:29.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:48:29.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:48:29.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:48:29.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:48:29.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:48:29.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:48:29.915 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T06:48:29.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:48:29.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:48:29.915 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:48:30.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149699 2024-02-05T06:48:30.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149699 2024-02-05T06:48:30.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149699' 2024-02-05T06:48:30.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:48:30.074 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:48:30.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-02-05T06:48:30.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-02-05T06:48:30.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149699 1-124554051586' 2024-02-05T06:48:30.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:48:30.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149699 2024-02-05T06:48:30.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:48:30.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:48:30.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149699 2024-02-05T06:48:30.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:48:30.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149699 2024-02-05T06:48:30.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2024-02-05T06:48:30.230 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 111669149699 2024-02-05T06:48:30.230 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:48:30.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149699 2024-02-05T06:48:30.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:48:31.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:48:31.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:48:31.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149699 -lt 111669149699 2024-02-05T06:48:31.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:48:31.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-124554051586 2024-02-05T06:48:31.871 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:48:31.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:48:31.874 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:48:31.874 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-124554051586 2024-02-05T06:48:31.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-02-05T06:48:31.875 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 124554051586 2024-02-05T06:48:31.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-02-05T06:48:31.876 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:48:32.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-02-05T06:48:32.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:48:32.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:48:32.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:48:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:48:32.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:48:32.582 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:48:32.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:48:32.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:48:32.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:48:32.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:48:32.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:48:32.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:48:32.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:48:32.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:48:32.896 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:48:33.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:48:33.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:48:33.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:48:33.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: local pgid=1.0 2024-02-05T06:48:33.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: pg_scrub 1.0 2024-02-05T06:48:33.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-02-05T06:48:33.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-02-05T06:48:33.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:48:33.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:48:33.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:48:33.282 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:48:33.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-02-05T06:47:15.233985+0000 2024-02-05T06:48:33.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-02-05T06:48:33.917 INFO:tasks.workunit.client.0.smithi148.stderr:instructing pg 1.0 on osd.1 to scrub 2024-02-05T06:48:33.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-02-05T06:47:15.233985+0000 2024-02-05T06:48:33.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:48:33.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:47:15.233985+0000 2024-02-05T06:48:33.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:48:33.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:48:33.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:48:33.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:48:33.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:48:33.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:48:33.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:48:33.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:48:34.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:47:15.233985+0000 '>' 2024-02-05T06:47:15.233985+0000 2024-02-05T06:48:34.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:48:35.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:48:35.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:48:35.258 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:48:35.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:48:35.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:48:35.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:48:35.260 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:48:35.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:47:15.233985+0000 '>' 2024-02-05T06:47:15.233985+0000 2024-02-05T06:48:35.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:48:36.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:48:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:48:36.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:48:36.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:48:36.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:48:36.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:48:36.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:48:36.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:47:15.233985+0000 '>' 2024-02-05T06:47:15.233985+0000 2024-02-05T06:48:36.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:48:37.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:48:37.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:48:37.924 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:48:37.924 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:48:37.924 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:48:37.924 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:48:37.925 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:48:38.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:47:15.233985+0000 '>' 2024-02-05T06:47:15.233985+0000 2024-02-05T06:48:38.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:48:39.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:48:39.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:48:39.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:48:39.249 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:48:39.249 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:48:39.249 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:48:39.249 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:48:39.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:48:35.613909+0000 '>' 2024-02-05T06:47:15.233985+0000 2024-02-05T06:48:39.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:48:39.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:783: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:48:39.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:783: _scrub_snaps_multi: wc -l 2024-02-05T06:48:39.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:783: _scrub_snaps_multi: test 16 -gt 3 2024-02-05T06:48:39.590 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:784: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:48:39.590 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:784: _scrub_snaps_multi: wc -l 2024-02-05T06:48:39.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:784: _scrub_snaps_multi: test 16 -gt 3 2024-02-05T06:48:39.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: rados list-inconsistent-pg test 2024-02-05T06:48:39.632 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:788: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2024-02-05T06:48:39.661 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:788: _scrub_snaps_multi: test 1 = 1 2024-02-05T06:48:39.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:790: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2024-02-05T06:48:39.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:790: _scrub_snaps_multi: test 1.0 = 1.0 2024-02-05T06:48:39.690 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:792: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2024-02-05T06:48:39.723 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:48:39.723 INFO:tasks.workunit.client.0.smithi148.stdout: "epoch": 29, 2024-02-05T06:48:39.723 INFO:tasks.workunit.client.0.smithi148.stdout: "inconsistents": [ 2024-02-05T06:48:39.723 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.723 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.723 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj5", 2024-02-05T06:48:39.723 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.723 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.723 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:48:39.723 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 5 2024-02-05T06:48:39.723 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.723 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.724 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:48:39.724 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:48:39.724 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.724 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.724 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.724 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj5", 2024-02-05T06:48:39.724 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.724 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": 1, 2024-02-05T06:48:39.724 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1718170787, 2024-02-05T06:48:39.724 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.724 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.724 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.724 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.724 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'19", 2024-02-05T06:48:39.725 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'5", 2024-02-05T06:48:39.725 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4245.0:1", 2024-02-05T06:48:39.725 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 5, 2024-02-05T06:48:39.725 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.725 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:19.858354+0000", 2024-02-05T06:48:39.725 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:19.859066+0000", 2024-02-05T06:48:39.725 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.725 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.725 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.725 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.725 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.725 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.725 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.726 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x767ddbf9", 2024-02-05T06:48:39.726 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.726 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.726 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.726 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.726 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.726 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.726 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.726 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.726 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.726 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.726 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.726 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.726 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.727 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj5", 2024-02-05T06:48:39.728 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.728 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.728 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 2, 2024-02-05T06:48:39.728 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 20 2024-02-05T06:48:39.728 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.728 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.728 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:48:39.728 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:48:39.728 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.728 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.728 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.728 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj5", 2024-02-05T06:48:39.728 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.729 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": 2, 2024-02-05T06:48:39.729 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1718170787, 2024-02-05T06:48:39.729 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.729 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.729 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.729 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.729 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "18'41", 2024-02-05T06:48:39.729 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "17'20", 2024-02-05T06:48:39.729 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4293.0:1", 2024-02-05T06:48:39.729 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 20, 2024-02-05T06:48:39.729 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.729 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:23.105918+0000", 2024-02-05T06:48:39.729 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:23.106662+0000", 2024-02-05T06:48:39.730 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.730 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.730 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.730 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.730 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.730 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.730 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.730 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x5a010218", 2024-02-05T06:48:39.730 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.730 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.730 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.730 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.731 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.731 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.731 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.731 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.731 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.731 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.731 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.731 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.731 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.731 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.731 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:48:39.731 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.731 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.731 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.732 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.732 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.732 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.732 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256 2024-02-05T06:48:39.732 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.732 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.732 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.732 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.732 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.732 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj5", 2024-02-05T06:48:39.732 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.732 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.732 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 4, 2024-02-05T06:48:39.732 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 42 2024-02-05T06:48:39.733 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.733 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.733 INFO:tasks.workunit.client.0.smithi148.stdout: "size_mismatch" 2024-02-05T06:48:39.733 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.733 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:48:39.733 INFO:tasks.workunit.client.0.smithi148.stdout: "size_mismatch_info", 2024-02-05T06:48:39.733 INFO:tasks.workunit.client.0.smithi148.stdout: "obj_size_info_mismatch" 2024-02-05T06:48:39.733 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.733 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.733 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.733 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj5", 2024-02-05T06:48:39.733 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.734 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": 4, 2024-02-05T06:48:39.734 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1718170787, 2024-02-05T06:48:39.734 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.734 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.734 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.734 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.734 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "20'45", 2024-02-05T06:48:39.734 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "18'42", 2024-02-05T06:48:39.734 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4329.0:1", 2024-02-05T06:48:39.734 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 42, 2024-02-05T06:48:39.734 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 512, 2024-02-05T06:48:39.734 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:25.026905+0000", 2024-02-05T06:48:39.734 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:25.042349+0000", 2024-02-05T06:48:39.734 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.735 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.735 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.735 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.735 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.735 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.735 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.735 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x01a3a510", 2024-02-05T06:48:39.735 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.735 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.735 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.735 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.735 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.735 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.735 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.736 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.736 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.736 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.736 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.736 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.736 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.736 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.736 INFO:tasks.workunit.client.0.smithi148.stdout: "size_mismatch_info", 2024-02-05T06:48:39.736 INFO:tasks.workunit.client.0.smithi148.stdout: "obj_size_info_mismatch" 2024-02-05T06:48:39.736 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.736 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 4608, 2024-02-05T06:48:39.736 INFO:tasks.workunit.client.0.smithi148.stdout: "object_info": { 2024-02-05T06:48:39.736 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.736 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj5", 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": 4, 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1718170787, 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "20'45", 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "18'42", 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4329.0:1", 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 42, 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 512, 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:25.026905+0000", 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:25.042349+0000", 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.737 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x01a3a510", 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.738 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 512 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj4", 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 7, 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 4 2024-02-05T06:48:39.739 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj4", 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": 7, 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 2826278768, 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.740 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "23'51", 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'4", 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4242.0:1", 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 4, 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:19.724247+0000", 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:19.724968+0000", 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x767ddbf9", 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.741 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.742 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.743 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.743 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.743 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.743 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032 2024-02-05T06:48:39.743 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.743 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.743 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.743 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.743 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.743 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj5", 2024-02-05T06:48:39.743 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.743 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 7, 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 0 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: "missing", 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: "info_missing" 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: "info_missing" 2024-02-05T06:48:39.744 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1792 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj1", 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.745 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 18 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj1", 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1828249343, 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.746 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.747 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.747 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.747 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'18", 2024-02-05T06:48:39.747 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'1", 2024-02-05T06:48:39.747 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4290.0:1", 2024-02-05T06:48:39.747 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 18, 2024-02-05T06:48:39.747 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.747 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:22.863907+0000", 2024-02-05T06:48:39.747 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:22.958567+0000", 2024-02-05T06:48:39.747 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.747 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.747 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.747 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.748 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.748 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.748 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.748 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x5a010218", 2024-02-05T06:48:39.748 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.748 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.748 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.748 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.748 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.748 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.748 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.748 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.748 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: "missing" 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.749 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj10", 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 32 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.750 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj10", 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 718195851, 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'32", 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'10", 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4311.0:1", 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 32, 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:24.057971+0000", 2024-02-05T06:48:39.751 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:24.058700+0000", 2024-02-05T06:48:39.752 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.752 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.752 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.752 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.752 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.752 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.752 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.752 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x5a010218", 2024-02-05T06:48:39.752 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.752 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.752 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.752 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.752 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.752 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:48:39.753 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.754 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "????", 2024-02-05T06:48:39.754 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:48:39.754 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:48:39.754 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.754 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.754 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.754 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.754 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.754 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.754 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.754 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.754 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.754 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.755 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.756 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.756 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.756 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj11", 2024-02-05T06:48:39.756 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.756 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.756 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:48:39.756 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 34 2024-02-05T06:48:39.756 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.756 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.756 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:48:39.756 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.756 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:48:39.756 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.756 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.757 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj11", 2024-02-05T06:48:39.757 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.757 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:48:39.757 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 693400951, 2024-02-05T06:48:39.757 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.757 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.757 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.757 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.757 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'34", 2024-02-05T06:48:39.757 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'11", 2024-02-05T06:48:39.757 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4314.0:1", 2024-02-05T06:48:39.757 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 34, 2024-02-05T06:48:39.757 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.758 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:24.210679+0000", 2024-02-05T06:48:39.758 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:24.211458+0000", 2024-02-05T06:48:39.758 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.758 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.758 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.758 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.758 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.758 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.758 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.758 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x5a010218", 2024-02-05T06:48:39.758 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.758 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.758 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.758 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.759 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.759 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.759 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.759 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.759 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.759 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.759 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.759 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.759 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.759 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.759 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.759 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.759 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:48:39.759 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [] 2024-02-05T06:48:39.760 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.760 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.760 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.760 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.760 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.760 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.760 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.760 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.760 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:48:39.760 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:48:39.760 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.760 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:48:39.760 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.760 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj13", 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:48:39.761 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 38 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj13", 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 2087409765, 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.762 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.763 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'38", 2024-02-05T06:48:39.763 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'13", 2024-02-05T06:48:39.763 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4320.0:1", 2024-02-05T06:48:39.763 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 38, 2024-02-05T06:48:39.763 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.763 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:24.530108+0000", 2024-02-05T06:48:39.763 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:24.530899+0000", 2024-02-05T06:48:39.763 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.763 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.763 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.763 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.763 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.763 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.763 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.764 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x5a010218", 2024-02-05T06:48:39.764 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.764 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.764 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.764 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.764 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.764 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.764 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.764 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.764 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.764 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.765 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.765 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.765 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.765 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.765 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.765 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.765 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:48:39.765 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:48:39.765 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.765 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:48:39.765 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.765 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]" 2024-02-05T06:48:39.765 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.765 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:48:39.766 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj14", 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:48:39.767 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 40 2024-02-05T06:48:39.768 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.768 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.768 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:48:39.768 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.768 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:48:39.768 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.768 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.768 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj14", 2024-02-05T06:48:39.768 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.768 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:48:39.768 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 2484217095, 2024-02-05T06:48:39.768 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.768 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.768 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'40", 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'14", 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4323.0:1", 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 40, 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:24.679065+0000", 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:24.679840+0000", 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.769 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x5a010218", 2024-02-05T06:48:39.770 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.770 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.770 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.770 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.770 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.770 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.770 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.770 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.770 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.770 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.770 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.770 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.770 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.770 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.771 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.771 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.771 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:48:39.771 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:48:39.771 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.771 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:48:39.771 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1033, 2024-02-05T06:48:39.771 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:48:39.771 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:48:39.771 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:48:39.771 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.771 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.771 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.771 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.772 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.772 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.772 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.772 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.772 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.772 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.772 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.772 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:48:39.772 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:48:39.772 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.772 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:48:39.772 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.772 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:48:39.773 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:48:39.773 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:48:39.773 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.773 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.773 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.773 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.773 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.773 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.773 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.773 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.773 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.773 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj15", 2024-02-05T06:48:39.773 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 15 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_corrupted" 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj15", 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 612772309, 2024-02-05T06:48:39.774 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.775 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.775 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.775 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.775 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "16'15", 2024-02-05T06:48:39.775 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "0'0", 2024-02-05T06:48:39.775 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4275.0:1", 2024-02-05T06:48:39.775 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 15, 2024-02-05T06:48:39.775 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.775 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:21.348817+0000", 2024-02-05T06:48:39.775 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:21.349585+0000", 2024-02-05T06:48:39.775 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.775 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.775 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.775 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.776 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.776 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.776 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.776 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x767ddbf9", 2024-02-05T06:48:39.776 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.776 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.776 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.776 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.776 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.776 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.776 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.776 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_corrupted" 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": "Z2FyYmFnZQo=" 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.777 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 0, 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [] 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj2", 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:48:39.778 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 56 2024-02-05T06:48:39.779 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.779 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.779 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:48:39.779 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_missing" 2024-02-05T06:48:39.779 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.779 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.779 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.779 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj2", 2024-02-05T06:48:39.779 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.779 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:48:39.779 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1058988552, 2024-02-05T06:48:39.779 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.779 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.779 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.780 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.780 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "23'56", 2024-02-05T06:48:39.780 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "20'48", 2024-02-05T06:48:39.780 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4365.0:1", 2024-02-05T06:48:39.780 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 56, 2024-02-05T06:48:39.780 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 0, 2024-02-05T06:48:39.780 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:27.825593+0000", 2024-02-05T06:48:39.780 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:27.826449+0000", 2024-02-05T06:48:39.780 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.780 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.780 INFO:tasks.workunit.client.0.smithi148.stdout: "whiteout", 2024-02-05T06:48:39.780 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty" 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0xffffffff", 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.781 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.782 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.782 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.782 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.782 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_missing" 2024-02-05T06:48:39.782 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.782 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 0 2024-02-05T06:48:39.782 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.782 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.782 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.782 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.782 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.782 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 0, 2024-02-05T06:48:39.782 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.782 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 7, 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 4, 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: 4, 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: 3, 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: 2, 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 7, 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1024, 2024-02-05T06:48:39.783 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:48:39.784 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:48:39.784 INFO:tasks.workunit.client.0.smithi148.stdout: 7, 2024-02-05T06:48:39.784 INFO:tasks.workunit.client.0.smithi148.stdout: 6, 2024-02-05T06:48:39.784 INFO:tasks.workunit.client.0.smithi148.stdout: 5 2024-02-05T06:48:39.784 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.784 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.784 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.784 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.784 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.784 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.784 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.784 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.785 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.785 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj3", 2024-02-05T06:48:39.785 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.785 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.785 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:48:39.785 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 44 2024-02-05T06:48:39.785 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.785 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.785 INFO:tasks.workunit.client.0.smithi148.stdout: "size_mismatch" 2024-02-05T06:48:39.785 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.785 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [ 2024-02-05T06:48:39.785 INFO:tasks.workunit.client.0.smithi148.stdout: "size_mismatch_info", 2024-02-05T06:48:39.785 INFO:tasks.workunit.client.0.smithi148.stdout: "obj_size_info_mismatch" 2024-02-05T06:48:39.785 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj3", 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1643547569, 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "19'44", 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "17'22", 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4335.0:1", 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 44, 2024-02-05T06:48:39.786 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 768, 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:25.547070+0000", 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:25.553240+0000", 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x32cec082", 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.787 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.788 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.788 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.788 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.788 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.788 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.788 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.788 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.788 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.788 INFO:tasks.workunit.client.0.smithi148.stdout: "size_mismatch_info", 2024-02-05T06:48:39.788 INFO:tasks.workunit.client.0.smithi148.stdout: "obj_size_info_mismatch" 2024-02-05T06:48:39.788 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.788 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 3840, 2024-02-05T06:48:39.788 INFO:tasks.workunit.client.0.smithi148.stdout: "object_info": { 2024-02-05T06:48:39.789 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.789 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj3", 2024-02-05T06:48:39.789 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.789 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:48:39.789 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1643547569, 2024-02-05T06:48:39.789 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.789 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.789 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.789 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.789 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "19'44", 2024-02-05T06:48:39.789 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "17'22", 2024-02-05T06:48:39.789 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4335.0:1", 2024-02-05T06:48:39.789 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 44, 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 768, 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:25.547070+0000", 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:25.553240+0000", 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x32cec082", 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.790 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 768 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.791 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.792 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj6", 2024-02-05T06:48:39.792 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.792 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.792 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:48:39.792 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 24 2024-02-05T06:48:39.792 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.792 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.792 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:48:39.792 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.792 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:48:39.792 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.792 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.792 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj6", 2024-02-05T06:48:39.792 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 2202164420, 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'24", 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'6", 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4299.0:1", 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 24, 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:23.433592+0000", 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:23.434347+0000", 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.793 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x5a010218", 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.794 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [] 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.795 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.796 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj7", 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 26 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.797 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj7", 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 1552453721, 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'26", 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'7", 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4302.0:1", 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 26, 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:23.590124+0000", 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:23.590865+0000", 2024-02-05T06:48:39.798 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x5a010218", 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.799 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 0, 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [] 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.800 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.801 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.801 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:48:39.801 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:48:39.801 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.801 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:48:39.801 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.801 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:48:39.801 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:48:39.801 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:48:39.801 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.801 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.801 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.801 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj8", 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 28 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.802 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:48:39.803 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.803 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.803 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj8", 2024-02-05T06:48:39.803 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.803 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:48:39.803 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 2381834917, 2024-02-05T06:48:39.803 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.803 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.803 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.803 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.803 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'28", 2024-02-05T06:48:39.803 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'8", 2024-02-05T06:48:39.803 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4305.0:1", 2024-02-05T06:48:39.803 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 28, 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:23.741395+0000", 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:23.742088+0000", 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x5a010218", 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.804 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 0, 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:48:39.805 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.806 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.807 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.807 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:48:39.807 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:48:39.807 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.807 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:48:39.807 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.807 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:48:39.807 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:48:39.807 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:48:39.807 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.807 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.807 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.807 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.807 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.808 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.808 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.808 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.808 INFO:tasks.workunit.client.0.smithi148.stdout: "object": { 2024-02-05T06:48:39.808 INFO:tasks.workunit.client.0.smithi148.stdout: "name": "obj9", 2024-02-05T06:48:39.808 INFO:tasks.workunit.client.0.smithi148.stdout: "nspace": "", 2024-02-05T06:48:39.808 INFO:tasks.workunit.client.0.smithi148.stdout: "locator": "", 2024-02-05T06:48:39.808 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": "head", 2024-02-05T06:48:39.808 INFO:tasks.workunit.client.0.smithi148.stdout: "version": 30 2024-02-05T06:48:39.808 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.808 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [ 2024-02-05T06:48:39.808 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset_inconsistency" 2024-02-05T06:48:39.808 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.809 INFO:tasks.workunit.client.0.smithi148.stdout: "union_shard_errors": [], 2024-02-05T06:48:39.809 INFO:tasks.workunit.client.0.smithi148.stdout: "selected_object_info": { 2024-02-05T06:48:39.809 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": { 2024-02-05T06:48:39.809 INFO:tasks.workunit.client.0.smithi148.stdout: "oid": "obj9", 2024-02-05T06:48:39.809 INFO:tasks.workunit.client.0.smithi148.stdout: "key": "", 2024-02-05T06:48:39.809 INFO:tasks.workunit.client.0.smithi148.stdout: "snapid": -2, 2024-02-05T06:48:39.809 INFO:tasks.workunit.client.0.smithi148.stdout: "hash": 3833113727, 2024-02-05T06:48:39.809 INFO:tasks.workunit.client.0.smithi148.stdout: "max": 0, 2024-02-05T06:48:39.809 INFO:tasks.workunit.client.0.smithi148.stdout: "pool": 1, 2024-02-05T06:48:39.809 INFO:tasks.workunit.client.0.smithi148.stdout: "namespace": "" 2024-02-05T06:48:39.809 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.809 INFO:tasks.workunit.client.0.smithi148.stdout: "version": "17'30", 2024-02-05T06:48:39.809 INFO:tasks.workunit.client.0.smithi148.stdout: "prior_version": "16'9", 2024-02-05T06:48:39.809 INFO:tasks.workunit.client.0.smithi148.stdout: "last_reqid": "client.4308.0:1", 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: "user_version": 30, 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: "mtime": "2024-02-05T06:47:23.909211+0000", 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: "local_mtime": "2024-02-05T06:47:23.909974+0000", 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: "lost": 0, 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: "flags": [ 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: "dirty", 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest" 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_seq": 0, 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: "truncate_size": 0, 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: "data_digest": "0x5a010218", 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_object_size": 0, 2024-02-05T06:48:39.810 INFO:tasks.workunit.client.0.smithi148.stdout: "expected_write_size": 0, 2024-02-05T06:48:39.811 INFO:tasks.workunit.client.0.smithi148.stdout: "alloc_hint_flags": 0, 2024-02-05T06:48:39.811 INFO:tasks.workunit.client.0.smithi148.stdout: "manifest": { 2024-02-05T06:48:39.811 INFO:tasks.workunit.client.0.smithi148.stdout: "type": 0 2024-02-05T06:48:39.811 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.811 INFO:tasks.workunit.client.0.smithi148.stdout: "watchers": {} 2024-02-05T06:48:39.811 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.811 INFO:tasks.workunit.client.0.smithi148.stdout: "shards": [ 2024-02-05T06:48:39.811 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.811 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 0, 2024-02-05T06:48:39.811 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": false, 2024-02-05T06:48:39.811 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.811 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.811 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.812 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:48:39.812 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:48:39.812 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.812 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:48:39.812 INFO:tasks.workunit.client.0.smithi148.stdout: "size": "????", 2024-02-05T06:48:39.812 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:48:39.812 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:48:39.812 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:48:39.812 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.812 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: "osd": 1, 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: "primary": true, 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: "errors": [], 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 256, 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: "snapset": { 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: "seq": 1, 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: "clones": [ 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: "snap": 1, 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: "size": 1032, 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: "overlap": "[]", 2024-02-05T06:48:39.813 INFO:tasks.workunit.client.0.smithi148.stdout: "snaps": [ 2024-02-05T06:48:39.814 INFO:tasks.workunit.client.0.smithi148.stdout: 1 2024-02-05T06:48:39.814 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.814 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.814 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.814 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.814 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.814 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.814 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:48:39.814 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:48:39.814 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:48:39.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:794: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2024-02-05T06:48:39.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: '[' replica = replica ']' 2024-02-05T06:48:39.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:799: _scrub_snaps_multi: scruberrors=20 2024-02-05T06:48:39.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: jq .inconsistents 2024-02-05T06:48:39.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:48:39.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1063: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2024-02-05T06:48:39.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1063: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:48:39.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1064: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:48:39.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:48:39.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1065: _scrub_snaps_multi: test no = yes 2024-02-05T06:48:39.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1070: _scrub_snaps_multi: test '' = yes 2024-02-05T06:48:39.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1075: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2024-02-05T06:48:39.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1075: _scrub_snaps_multi: find td/osd-scrub-snaps 2024-02-05T06:48:39.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1075: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2024-02-05T06:48:39.826 INFO:tasks.workunit.client.0.smithi148.stderr:td/osd-scrub-snaps/osd.0.pid' 2024-02-05T06:48:39.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1076: _scrub_snaps_multi: pids= 2024-02-05T06:48:39.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1077: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-02-05T06:48:39.826 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2024-02-05T06:48:39.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids+='543736 ' 2024-02-05T06:48:39.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1077: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-02-05T06:48:39.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2024-02-05T06:48:39.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids+='543172 ' 2024-02-05T06:48:39.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: ERRORS=0 2024-02-05T06:48:39.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1086: _scrub_snaps_multi: '[' replica = primary ']' 2024-02-05T06:48:39.828 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1108: _scrub_snaps_multi: ceph pg dump pgs 2024-02-05T06:48:40.115 INFO:tasks.workunit.client.0.smithi148.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T06:48:40.115 INFO:tasks.workunit.client.0.smithi148.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent 2024-02-05T06:48:35.614015+0000 23'56 30:95 [1,0] 1 [1,0] 1 23'56 2024-02-05T06:48:35.613909+0000 0'0 2024-02-05T06:47:15.233985+0000 0 1 periodic scrub scheduled @ 2024-02-06T08:20:52.606329+0000 36 0 2024-02-05T06:48:40.115 INFO:tasks.workunit.client.0.smithi148.stdout: 2024-02-05T06:48:40.115 INFO:tasks.workunit.client.0.smithi148.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T06:48:40.115 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:48:40.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1110: _scrub_snaps_multi: for pid in $pids 2024-02-05T06:48:40.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1112: _scrub_snaps_multi: kill -0 543736 2024-02-05T06:48:40.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1110: _scrub_snaps_multi: for pid in $pids 2024-02-05T06:48:40.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1112: _scrub_snaps_multi: kill -0 543172 2024-02-05T06:48:40.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2024-02-05T06:48:40.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:48:40.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:48:40.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:48:40.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:48:40.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:48:45.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:48:45.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1121: _scrub_snaps_multi: declare -a err_strings 2024-02-05T06:48:45.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2024-02-05T06:48:45.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1123: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2024-02-05T06:48:45.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2024-02-05T06:48:45.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2024-02-05T06:48:45.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2024-02-05T06:48:45.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2024-02-05T06:48:45.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2024-02-05T06:48:45.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2024-02-05T06:48:45.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:48:45.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:48:45.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:48:45.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:48:45.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:48:45.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:48:45.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:48:45.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:48:45.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:48:45.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:48:45.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:48:45.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:48:45.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:48:45.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:48:45.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:48:45.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:48:45.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1141: _scrub_snaps_multi: declare -a rep_err_strings 2024-02-05T06:48:45.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1142: _scrub_snaps_multi: eval echo '$replica' 2024-02-05T06:48:45.481 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1142: _scrub_snaps_multi: echo 0 2024-02-05T06:48:45.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1142: _scrub_snaps_multi: osd=0 2024-02-05T06:48:45.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1143: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-02-05T06:48:45.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2024-02-05T06:48:45.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:48:45.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1153: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2024-02-05T06:48:45.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1159: _scrub_snaps_multi: echo 'TEST PASSED' 2024-02-05T06:48:45.485 INFO:tasks.workunit.client.0.smithi148.stdout:TEST PASSED 2024-02-05T06:48:45.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1160: _scrub_snaps_multi: return 0 2024-02-05T06:48:45.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1168: TEST_scrub_snaps_replica: err=0 2024-02-05T06:48:45.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1169: TEST_scrub_snaps_replica: CEPH_ARGS='--fsid=7a42756a-53bc-4a19-ad97-d3ea7c54b526 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-02-05T06:48:45.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1170: TEST_scrub_snaps_replica: return 0 2024-02-05T06:48:45.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-02-05T06:48:45.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-02-05T06:48:45.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:48:45.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-02-05T06:48:45.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:48:45.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:48:45.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:48:45.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:48:45.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:48:45.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:48:45.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:48:45.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:48:45.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:48:45.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:48:45.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:48:45.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:48:45.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:48:45.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:48:45.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:48:45.498 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:48:45.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:48:45.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:48:45.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-02-05T06:48:45.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:48:45.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:48:45.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:48:45.518 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.479281 2024-02-05T06:48:45.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:48:45.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:48:45.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-05T06:48:45.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-snaps 0 2024-02-05T06:48:45.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-02-05T06:48:45.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-05T06:48:45.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-02-05T06:48:45.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:48:45.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:48:45.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:48:45.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:48:45.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:48:45.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:48:45.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:48:45.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:48:45.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:48:45.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:48:45.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:48:45.526 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:48:45.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:48:45.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:48:45.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:48:45.527 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:48:45.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:48:45.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-05T06:48:45.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-02-05T06:48:45.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:48:45.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:48:45.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.479281 2024-02-05T06:48:45.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.479281 2024-02-05T06:48:45.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:48:45.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:48:45.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-05T06:48:45.533 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-05T06:48:45.534 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-05T06:48:45.545 INFO:tasks.workunit:Running workunit scrub/osd-scrub-test.sh... 2024-02-05T06:48:45.545 DEBUG:teuthology.orchestra.run.smithi148:workunit test scrub/osd-scrub-test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh 2024-02-05T06:48:45.598 INFO:tasks.workunit.client.0.smithi148.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-05T06:48:45.604 INFO:tasks.workunit.client.0.smithi148.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-05T06:48:45.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-05T06:48:45.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-02-05T06:48:45.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-05T06:48:45.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-05T06:48:45.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-05T06:48:45.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-05T06:48:45.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-05T06:48:45.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-05T06:48:45.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-test 2024-02-05T06:48:45.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:21: run: local dir=td/osd-scrub-test 2024-02-05T06:48:45.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:22: run: shift 2024-02-05T06:48:45.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: export CEPH_MON=127.0.0.1:7138 2024-02-05T06:48:45.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: CEPH_MON=127.0.0.1:7138 2024-02-05T06:48:45.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:25: run: export CEPH_ARGS 2024-02-05T06:48:45.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: uuidgen 2024-02-05T06:48:45.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: CEPH_ARGS+='--fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none ' 2024-02-05T06:48:45.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7138 ' 2024-02-05T06:48:45.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-02-05T06:48:45.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: set 2024-02-05T06:48:45.607 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-05T06:48:45.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: local 'funcs=TEST_deep_scrub_abort 2024-02-05T06:48:45.608 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_dump_scrub_schedule 2024-02-05T06:48:45.608 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_interval_changes 2024-02-05T06:48:45.608 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_just_deep_scrubs 2024-02-05T06:48:45.608 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_pg_dump_objects_scrubbed 2024-02-05T06:48:45.608 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_scrub_abort 2024-02-05T06:48:45.609 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_scrub_extended_sleep 2024-02-05T06:48:45.609 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_scrub_permit_time 2024-02-05T06:48:45.609 INFO:tasks.workunit.client.0.smithi148.stderr:TEST_scrub_test' 2024-02-05T06:48:45.609 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-02-05T06:48:45.609 INFO:tasks.workunit.client.0.smithi148.stdout:-------------- Prepare Test TEST_deep_scrub_abort ------------------- 2024-02-05T06:48:45.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_deep_scrub_abort -------------------' 2024-02-05T06:48:45.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-02-05T06:48:45.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-02-05T06:48:45.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-02-05T06:48:45.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T06:48:45.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:48:45.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T06:48:45.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:48:45.610 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:48:45.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:48:45.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:48:45.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:48:45.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:48:45.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:48:45.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:48:45.612 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:48:45.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:48:45.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:48:45.613 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:48:45.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:48:45.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:48:45.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:48:45.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:48:45.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:48:45.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:48:45.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T06:48:45.618 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:48:45.618 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:48:45.618 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:48:45.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.545011 2024-02-05T06:48:45.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:48:45.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:48:45.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-02-05T06:48:45.621 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:48:45.621 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:48:45.621 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:48:45.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.545011 2024-02-05T06:48:45.623 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:48:45.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-05T06:48:45.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-05T06:48:45.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:48:45.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-02-05T06:48:45.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_deep_scrub_abort -----------------------' 2024-02-05T06:48:45.623 INFO:tasks.workunit.client.0.smithi148.stdout:-------------- Run Test TEST_deep_scrub_abort ----------------------- 2024-02-05T06:48:45.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_deep_scrub_abort td/osd-scrub-test 2024-02-05T06:48:45.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:414: TEST_deep_scrub_abort: local dir=td/osd-scrub-test 2024-02-05T06:48:45.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:415: TEST_deep_scrub_abort: _scrub_abort td/osd-scrub-test deep-scrub 2024-02-05T06:48:45.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:298: _scrub_abort: local dir=td/osd-scrub-test 2024-02-05T06:48:45.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:299: _scrub_abort: local poolname=test 2024-02-05T06:48:45.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: local OSDS=3 2024-02-05T06:48:45.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:301: _scrub_abort: local objects=1000 2024-02-05T06:48:45.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:302: _scrub_abort: local type=deep-scrub 2024-02-05T06:48:45.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:304: _scrub_abort: TESTDATA=testdata.545011 2024-02-05T06:48:45.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:305: _scrub_abort: test deep-scrub = scrub 2024-02-05T06:48:45.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: stopscrub=nodeep-scrub 2024-02-05T06:48:45.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: check=nodeep_scrub 2024-02-05T06:48:45.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:314: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-02-05T06:48:45.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-02-05T06:48:45.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:48:45.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:48:45.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:48:45.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-02-05T06:48:45.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-02-05T06:48:46.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:48:46.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:48:46.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:48:46.026 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:48:46.027 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:48:46.027 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:48:46.027 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:48:46.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-02-05T06:48:46.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:48:46.238 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:48:46.238 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:48:46.238 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:48:46.238 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:48:46.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:48:46.240 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:48:46.240 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:48:46.240 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:48:46.240 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:48:46.240 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:48:46.240 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:48:46.240 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.545011/ceph-mon.a.asok 2024-02-05T06:48:46.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:48:46.240 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.545011/ceph-mon.a.asok config get fsid 2024-02-05T06:48:46.351 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:48:46.352 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:48:46.352 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:48:46.352 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:48:46.352 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:48:46.353 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:48:46.353 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:48:46.353 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:48:46.353 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:48:46.353 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:48:46.353 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:48:46.353 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.545011/ceph-mon.a.asok 2024-02-05T06:48:46.354 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:48:46.354 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.545011/ceph-mon.a.asok config get mon_host 2024-02-05T06:48:46.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: run_mgr td/osd-scrub-test x 2024-02-05T06:48:46.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-02-05T06:48:46.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:48:46.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:48:46.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:48:46.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-02-05T06:48:46.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:48:47.338 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:48:47.338 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:48:47.338 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:48:47.338 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:48:47.338 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:48:47.338 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:48:47.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:48:47.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:48:47.341 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:48:47.373 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: expr 3 - 1 2024-02-05T06:48:47.373 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: seq 0 2 2024-02-05T06:48:47.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:48:47.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:322: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-02-05T06:48:47.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:48:47.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:48:47.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:48:47.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:48:47.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-02-05T06:48:47.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:48:47.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:48:47.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:48:47.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:48:47.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-02-05T06:48:47.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-02-05T06:48:47.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:48:47.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:48:47.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:48:47.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:48:47.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:48:47.378 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:48:47.378 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:48:47.378 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:48:47.378 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:48:47.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:48:47.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:48:47.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:48:47.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:48:47.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:48:47.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:48:47.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:48:47.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:48:47.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:48:47.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:48:47.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:48:47.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:48:47.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-02-05T06:48:47.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-02-05T06:48:47.380 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:48:47.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c4e2a111-066a-42ca-9126-294889f2d04a 2024-02-05T06:48:47.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c4e2a111-066a-42ca-9126-294889f2d04a' 2024-02-05T06:48:47.380 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 c4e2a111-066a-42ca-9126-294889f2d04a 2024-02-05T06:48:47.381 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:48:47.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPhMBlDY4bFxAAuXrqWVsoB0gyNW51APeSSA== 2024-02-05T06:48:47.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPhMBlDY4bFxAAuXrqWVsoB0gyNW51APeSSA=="}' 2024-02-05T06:48:47.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c4e2a111-066a-42ca-9126-294889f2d04a -i td/osd-scrub-test/0/new.json 2024-02-05T06:48:47.649 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:48:47.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-02-05T06:48:47.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQDPhMBlDY4bFxAAuXrqWVsoB0gyNW51APeSSA== --osd-uuid c4e2a111-066a-42ca-9126-294889f2d04a 2024-02-05T06:48:47.703 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:47.693+0000 7f48bc9db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:47.705 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:47.697+0000 7f48bc9db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:47.707 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:47.697+0000 7f48bc9db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:47.707 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:47.697+0000 7f48bc9db640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:48:47.707 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:47.697+0000 7f48bc9db640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:48:47.707 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:47.697+0000 7f48bc9db640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:48:47.792 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:47.781+0000 7f48bc9db640 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-02-05T06:48:54.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-02-05T06:48:54.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:48:54.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:48:54.032 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:48:54.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:48:54.396 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:48:54.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:48:54.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:48:54.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-02-05T06:48:54.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:48:54.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:48:54.431 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:54.421+0000 7fe06d47f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:54.465 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:54.457+0000 7fe06d47f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:54.467 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:54.457+0000 7fe06d47f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:54.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:48:54.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:48:54.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:48:54.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:48:54.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:48:54.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:48:54.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:48:54.692 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:48:54.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:48:54.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:48:55.026 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:55.017+0000 7fe06d47f640 -1 Falling back to public interface 2024-02-05T06:48:55.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:48:56.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:48:56.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:48:56.029 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:48:56.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:48:56.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:48:56.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:48:56.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:48:57.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:48:57.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:48:57.331 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:48:57.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:48:57.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:48:57.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:48:57.601 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:57.593+0000 7fe06d47f640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:48:57.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:48:58.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:48:58.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:48:58.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:48:58.638 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:48:58.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:48:58.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:48:58.951 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3858121790,v1:127.0.0.1:6803/3858121790] [v2:127.0.0.1:6804/3858121790,v1:127.0.0.1:6805/3858121790] exists,up c4e2a111-066a-42ca-9126-294889f2d04a 2024-02-05T06:48:58.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:48:58.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:48:58.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:48:58.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:48:58.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:322: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-02-05T06:48:58.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:48:58.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:48:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:48:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:48:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-02-05T06:48:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:48:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:48:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:48:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:48:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-02-05T06:48:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-02-05T06:48:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:48:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:48:58.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:48:58.960 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:48:58.960 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:48:58.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:48:58.961 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:48:58.961 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:48:58.961 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:48:58.961 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:48:58.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:48:58.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:48:58.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:48:58.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:48:58.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:48:58.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:48:58.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:48:58.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:48:58.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:48:58.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:48:58.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:48:58.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-02-05T06:48:58.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-02-05T06:48:58.962 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:48:58.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=838d7acf-408d-4403-bf83-6eb25472d152 2024-02-05T06:48:58.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 838d7acf-408d-4403-bf83-6eb25472d152' 2024-02-05T06:48:58.962 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 838d7acf-408d-4403-bf83-6eb25472d152 2024-02-05T06:48:58.962 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:48:58.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDahMBlWMB+ORAAYzz6ygByD3xTnWWrJ3ws1Q== 2024-02-05T06:48:58.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDahMBlWMB+ORAAYzz6ygByD3xTnWWrJ3ws1Q=="}' 2024-02-05T06:48:58.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 838d7acf-408d-4403-bf83-6eb25472d152 -i td/osd-scrub-test/1/new.json 2024-02-05T06:48:59.319 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:48:59.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-02-05T06:48:59.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQDahMBlWMB+ORAAYzz6ygByD3xTnWWrJ3ws1Q== --osd-uuid 838d7acf-408d-4403-bf83-6eb25472d152 2024-02-05T06:48:59.368 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:59.357+0000 7f3b7af86640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:59.378 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:59.369+0000 7f3b7af86640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:59.380 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:59.369+0000 7f3b7af86640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:48:59.380 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:59.373+0000 7f3b7af86640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:48:59.380 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:59.373+0000 7f3b7af86640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:48:59.380 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:59.373+0000 7f3b7af86640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:48:59.454 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:48:59.445+0000 7f3b7af86640 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-02-05T06:49:05.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-02-05T06:49:05.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:49:05.632 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:49:05.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:49:05.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:49:06.007 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:49:06.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:49:06.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:49:06.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:49:06.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:49:06.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-02-05T06:49:06.040 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:06.033+0000 7fe691082640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:49:06.076 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:06.069+0000 7fe691082640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:49:06.078 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:06.069+0000 7fe691082640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:49:06.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:49:06.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:49:06.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:49:06.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:49:06.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:49:06.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:49:06.354 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:49:06.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:49:06.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:49:06.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:49:06.627 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:06.617+0000 7fe691082640 -1 Falling back to public interface 2024-02-05T06:49:06.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:49:07.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:49:07.671 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:49:07.672 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:49:07.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:49:07.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:49:07.672 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:49:07.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:49:08.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:49:08.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:49:08.980 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:49:08.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:49:08.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:49:08.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:49:09.127 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:09.117+0000 7fe691082640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:49:09.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:49:10.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:49:10.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:49:10.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:49:10.363 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:49:10.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:49:10.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:49:10.697 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/918355953,v1:127.0.0.1:6811/918355953] [v2:127.0.0.1:6812/918355953,v1:127.0.0.1:6813/918355953] exists,up 838d7acf-408d-4403-bf83-6eb25472d152 2024-02-05T06:49:10.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:49:10.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:49:10.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:49:10.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:49:10.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:322: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-02-05T06:49:10.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:49:10.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:49:10.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T06:49:10.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:49:10.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/2 2024-02-05T06:49:10.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:49:10.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:49:10.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:49:10.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:49:10.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-02-05T06:49:10.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-02-05T06:49:10.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:49:10.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:49:10.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:49:10.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:49:10.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:49:10.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:49:10.701 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:49:10.701 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:49:10.701 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:49:10.701 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:49:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:49:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:49:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:49:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:49:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:49:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:49:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:49:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:49:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:49:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:49:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:49:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-02-05T06:49:10.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/2 2024-02-05T06:49:10.703 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:49:10.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=63f15da7-b762-4cfe-a4da-bae98a63c02b 2024-02-05T06:49:10.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 63f15da7-b762-4cfe-a4da-bae98a63c02b' 2024-02-05T06:49:10.704 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 63f15da7-b762-4cfe-a4da-bae98a63c02b 2024-02-05T06:49:10.704 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:49:10.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDmhMBlUPSjKhAApkEoidsMIzH1apAM/xJV4Q== 2024-02-05T06:49:10.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDmhMBlUPSjKhAApkEoidsMIzH1apAM/xJV4Q=="}' 2024-02-05T06:49:10.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 63f15da7-b762-4cfe-a4da-bae98a63c02b -i td/osd-scrub-test/2/new.json 2024-02-05T06:49:11.098 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:49:11.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/2/new.json 2024-02-05T06:49:11.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQDmhMBlUPSjKhAApkEoidsMIzH1apAM/xJV4Q== --osd-uuid 63f15da7-b762-4cfe-a4da-bae98a63c02b 2024-02-05T06:49:11.145 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:11.137+0000 7ff9bdace640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:49:11.191 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:11.181+0000 7ff9bdace640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:49:11.193 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:11.185+0000 7ff9bdace640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:49:11.194 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:11.185+0000 7ff9bdace640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:49:11.194 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:11.185+0000 7ff9bdace640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:49:11.194 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:11.185+0000 7ff9bdace640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:49:11.272 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:11.265+0000 7ff9bdace640 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-02-05T06:49:17.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-02-05T06:49:17.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:49:17.947 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T06:49:17.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T06:49:17.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:49:18.313 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T06:49:18.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T06:49:18.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-02-05T06:49:18.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:49:18.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:49:18.314 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:49:18.347 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:18.336+0000 7f5d6a6c7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:49:18.373 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:18.364+0000 7f5d6a6c7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:49:18.374 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:18.364+0000 7f5d6a6c7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:49:18.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T06:49:18.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:49:18.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:49:18.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:49:18.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:49:18.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:49:18.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:49:18.634 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:49:18.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:49:18.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:49:18.926 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:18.916+0000 7f5d6a6c7640 -1 Falling back to public interface 2024-02-05T06:49:18.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:49:19.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:49:19.963 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:49:19.964 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:49:19.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:49:19.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:49:19.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:49:20.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:49:21.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:49:21.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:49:21.269 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:49:21.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:49:21.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:49:21.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:49:21.575 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:49:21.588 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:49:21.576+0000 7f5d6a6c7640 -1 osd.2 0 log_to_monitors true 2024-02-05T06:49:22.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:49:22.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:49:22.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:49:22.577 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:49:22.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:49:22.578 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:49:22.891 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4038981208,v1:127.0.0.1:6819/4038981208] [v2:127.0.0.1:6820/4038981208,v1:127.0.0.1:6821/4038981208] exists,up 63f15da7-b762-4cfe-a4da-bae98a63c02b 2024-02-05T06:49:22.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:49:22.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:49:22.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:49:22.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: create_pool test 1 1 2024-02-05T06:49:22.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-05T06:49:23.313 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'test' created 2024-02-05T06:49:23.336 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:49:24.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:331: _scrub_abort: wait_for_clean 2024-02-05T06:49:24.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:49:24.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:49:24.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:49:24.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:49:24.340 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:49:24.340 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:49:24.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:49:24.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:49:24.342 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:49:24.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:49:24.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:49:24.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:49:24.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:49:24.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:49:24.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:49:24.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:49:24.858 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:49:24.858 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:49:24.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:49:24.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:49:24.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:49:25.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-02-05T06:49:25.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-02-05T06:49:25.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-02-05T06:49:25.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:49:25.004 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:49:25.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-02-05T06:49:25.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-02-05T06:49:25.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2024-02-05T06:49:25.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:49:25.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:49:25.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-02-05T06:49:25.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-02-05T06:49:25.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2024-02-05T06:49:25.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:49:25.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-02-05T06:49:25.340 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:49:25.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:49:25.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-02-05T06:49:25.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:49:25.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-02-05T06:49:25.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-02-05T06:49:25.343 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836487 2024-02-05T06:49:25.343 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:49:25.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836487 2024-02-05T06:49:25.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:49:26.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:49:26.633 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:49:26.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836487 2024-02-05T06:49:26.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:49:26.934 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-02-05T06:49:26.934 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:49:26.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:49:26.936 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-02-05T06:49:26.937 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:49:26.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-02-05T06:49:26.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-02-05T06:49:26.939 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672964 2024-02-05T06:49:26.940 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:49:27.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672964 2024-02-05T06:49:27.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:49:27.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-02-05T06:49:27.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:49:27.249 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:49:27.250 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-02-05T06:49:27.250 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:49:27.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-02-05T06:49:27.251 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509442 2024-02-05T06:49:27.251 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-02-05T06:49:27.251 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:49:27.572 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-02-05T06:49:27.573 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:49:27.573 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:49:27.573 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:49:27.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:49:27.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:49:27.989 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:49:27.989 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:49:27.989 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:49:27.989 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:49:27.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:49:27.990 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:49:28.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:49:28.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:49:28.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:49:28.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:49:28.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:49:28.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:49:28.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:49:28.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: grep '^pool.*['\'']test['\'']' 2024-02-05T06:49:28.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: ceph osd dump 2024-02-05T06:49:28.721 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: awk '{ print $2 }' 2024-02-05T06:49:29.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: poolid=1 2024-02-05T06:49:29.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: dd if=/dev/urandom of=testdata.545011 bs=1032 count=1 2024-02-05T06:49:29.048 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records in 2024-02-05T06:49:29.048 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records out 2024-02-05T06:49:29.048 INFO:tasks.workunit.client.0.smithi148.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000112333 s, 9.2 MB/s 2024-02-05T06:49:29.050 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: seq 1 1000 2024-02-05T06:49:29.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:29.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj1 testdata.545011 2024-02-05T06:49:29.228 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:29.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj2 testdata.545011 2024-02-05T06:49:29.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:29.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj3 testdata.545011 2024-02-05T06:49:29.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:29.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj4 testdata.545011 2024-02-05T06:49:29.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:29.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj5 testdata.545011 2024-02-05T06:49:29.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:29.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj6 testdata.545011 2024-02-05T06:49:30.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:30.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj7 testdata.545011 2024-02-05T06:49:30.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:30.173 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj8 testdata.545011 2024-02-05T06:49:30.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:30.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj9 testdata.545011 2024-02-05T06:49:30.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:30.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj10 testdata.545011 2024-02-05T06:49:30.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:30.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj11 testdata.545011 2024-02-05T06:49:30.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:30.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj12 testdata.545011 2024-02-05T06:49:31.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:31.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj13 testdata.545011 2024-02-05T06:49:31.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:31.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj14 testdata.545011 2024-02-05T06:49:31.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:31.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj15 testdata.545011 2024-02-05T06:49:31.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:31.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj16 testdata.545011 2024-02-05T06:49:31.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:31.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj17 testdata.545011 2024-02-05T06:49:31.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:31.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj18 testdata.545011 2024-02-05T06:49:31.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:31.973 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj19 testdata.545011 2024-02-05T06:49:32.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:32.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj20 testdata.545011 2024-02-05T06:49:32.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:32.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj21 testdata.545011 2024-02-05T06:49:32.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:32.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj22 testdata.545011 2024-02-05T06:49:32.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:32.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj23 testdata.545011 2024-02-05T06:49:32.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:32.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj24 testdata.545011 2024-02-05T06:49:33.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:33.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj25 testdata.545011 2024-02-05T06:49:33.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:33.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj26 testdata.545011 2024-02-05T06:49:33.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:33.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj27 testdata.545011 2024-02-05T06:49:33.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:33.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj28 testdata.545011 2024-02-05T06:49:33.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:33.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj29 testdata.545011 2024-02-05T06:49:33.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:33.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj30 testdata.545011 2024-02-05T06:49:34.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:34.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj31 testdata.545011 2024-02-05T06:49:34.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:34.168 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj32 testdata.545011 2024-02-05T06:49:34.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:34.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj33 testdata.545011 2024-02-05T06:49:34.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:34.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj34 testdata.545011 2024-02-05T06:49:34.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:34.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj35 testdata.545011 2024-02-05T06:49:34.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:34.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj36 testdata.545011 2024-02-05T06:49:35.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:35.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj37 testdata.545011 2024-02-05T06:49:35.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:35.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj38 testdata.545011 2024-02-05T06:49:35.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:35.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj39 testdata.545011 2024-02-05T06:49:35.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:35.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj40 testdata.545011 2024-02-05T06:49:35.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:35.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj41 testdata.545011 2024-02-05T06:49:35.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:35.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj42 testdata.545011 2024-02-05T06:49:35.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:35.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj43 testdata.545011 2024-02-05T06:49:36.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:36.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj44 testdata.545011 2024-02-05T06:49:36.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:36.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj45 testdata.545011 2024-02-05T06:49:36.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:36.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj46 testdata.545011 2024-02-05T06:49:36.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:36.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj47 testdata.545011 2024-02-05T06:49:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:36.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj48 testdata.545011 2024-02-05T06:49:37.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:37.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj49 testdata.545011 2024-02-05T06:49:37.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:37.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj50 testdata.545011 2024-02-05T06:49:37.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:37.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj51 testdata.545011 2024-02-05T06:49:37.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:37.501 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj52 testdata.545011 2024-02-05T06:49:37.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:37.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj53 testdata.545011 2024-02-05T06:49:37.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:37.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj54 testdata.545011 2024-02-05T06:49:38.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:38.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj55 testdata.545011 2024-02-05T06:49:38.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:38.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj56 testdata.545011 2024-02-05T06:49:38.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:38.315 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj57 testdata.545011 2024-02-05T06:49:38.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:38.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj58 testdata.545011 2024-02-05T06:49:38.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:38.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj59 testdata.545011 2024-02-05T06:49:38.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:38.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj60 testdata.545011 2024-02-05T06:49:39.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:39.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj61 testdata.545011 2024-02-05T06:49:39.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:39.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj62 testdata.545011 2024-02-05T06:49:39.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:39.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj63 testdata.545011 2024-02-05T06:49:39.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:39.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj64 testdata.545011 2024-02-05T06:49:39.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:39.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj65 testdata.545011 2024-02-05T06:49:39.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:39.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj66 testdata.545011 2024-02-05T06:49:40.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:40.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj67 testdata.545011 2024-02-05T06:49:40.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:40.233 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj68 testdata.545011 2024-02-05T06:49:40.392 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:40.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj69 testdata.545011 2024-02-05T06:49:40.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:40.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj70 testdata.545011 2024-02-05T06:49:40.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:40.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj71 testdata.545011 2024-02-05T06:49:40.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:40.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj72 testdata.545011 2024-02-05T06:49:41.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:41.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj73 testdata.545011 2024-02-05T06:49:41.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:41.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj74 testdata.545011 2024-02-05T06:49:41.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:41.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj75 testdata.545011 2024-02-05T06:49:41.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:41.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj76 testdata.545011 2024-02-05T06:49:41.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:41.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj77 testdata.545011 2024-02-05T06:49:41.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:41.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj78 testdata.545011 2024-02-05T06:49:42.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:42.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj79 testdata.545011 2024-02-05T06:49:42.193 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:42.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj80 testdata.545011 2024-02-05T06:49:42.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:42.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj81 testdata.545011 2024-02-05T06:49:42.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:42.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj82 testdata.545011 2024-02-05T06:49:42.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:42.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj83 testdata.545011 2024-02-05T06:49:42.905 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:42.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj84 testdata.545011 2024-02-05T06:49:43.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:43.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj85 testdata.545011 2024-02-05T06:49:43.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:43.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj86 testdata.545011 2024-02-05T06:49:43.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:43.391 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj87 testdata.545011 2024-02-05T06:49:43.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:43.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj88 testdata.545011 2024-02-05T06:49:43.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:43.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj89 testdata.545011 2024-02-05T06:49:43.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:43.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj90 testdata.545011 2024-02-05T06:49:44.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:44.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj91 testdata.545011 2024-02-05T06:49:44.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:44.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj92 testdata.545011 2024-02-05T06:49:44.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:44.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj93 testdata.545011 2024-02-05T06:49:44.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:44.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj94 testdata.545011 2024-02-05T06:49:44.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:44.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj95 testdata.545011 2024-02-05T06:49:44.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:44.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj96 testdata.545011 2024-02-05T06:49:45.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:45.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj97 testdata.545011 2024-02-05T06:49:45.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:45.191 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj98 testdata.545011 2024-02-05T06:49:45.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:45.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj99 testdata.545011 2024-02-05T06:49:45.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:45.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj100 testdata.545011 2024-02-05T06:49:45.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:45.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj101 testdata.545011 2024-02-05T06:49:45.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:45.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj102 testdata.545011 2024-02-05T06:49:46.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:46.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj103 testdata.545011 2024-02-05T06:49:46.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:46.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj104 testdata.545011 2024-02-05T06:49:46.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:46.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj105 testdata.545011 2024-02-05T06:49:46.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:46.515 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj106 testdata.545011 2024-02-05T06:49:46.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:46.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj107 testdata.545011 2024-02-05T06:49:46.891 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:46.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj108 testdata.545011 2024-02-05T06:49:47.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:47.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj109 testdata.545011 2024-02-05T06:49:47.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:47.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj110 testdata.545011 2024-02-05T06:49:47.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:47.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj111 testdata.545011 2024-02-05T06:49:47.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:47.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj112 testdata.545011 2024-02-05T06:49:47.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:47.721 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj113 testdata.545011 2024-02-05T06:49:47.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:47.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj114 testdata.545011 2024-02-05T06:49:48.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:48.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj115 testdata.545011 2024-02-05T06:49:48.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:48.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj116 testdata.545011 2024-02-05T06:49:48.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:48.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj117 testdata.545011 2024-02-05T06:49:48.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:48.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj118 testdata.545011 2024-02-05T06:49:48.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:48.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj119 testdata.545011 2024-02-05T06:49:48.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:48.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj120 testdata.545011 2024-02-05T06:49:49.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:49.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj121 testdata.545011 2024-02-05T06:49:49.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:49.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj122 testdata.545011 2024-02-05T06:49:49.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:49.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj123 testdata.545011 2024-02-05T06:49:49.825 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:49.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj124 testdata.545011 2024-02-05T06:49:49.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:49.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj125 testdata.545011 2024-02-05T06:49:50.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:50.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj126 testdata.545011 2024-02-05T06:49:50.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:50.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj127 testdata.545011 2024-02-05T06:49:50.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:50.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj128 testdata.545011 2024-02-05T06:49:50.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:50.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj129 testdata.545011 2024-02-05T06:49:50.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:50.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj130 testdata.545011 2024-02-05T06:49:50.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:50.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj131 testdata.545011 2024-02-05T06:49:51.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:51.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj132 testdata.545011 2024-02-05T06:49:51.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:51.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj133 testdata.545011 2024-02-05T06:49:51.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:51.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj134 testdata.545011 2024-02-05T06:49:51.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:51.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj135 testdata.545011 2024-02-05T06:49:51.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:51.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj136 testdata.545011 2024-02-05T06:49:51.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:51.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj137 testdata.545011 2024-02-05T06:49:52.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:52.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj138 testdata.545011 2024-02-05T06:49:52.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:52.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj139 testdata.545011 2024-02-05T06:49:52.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:52.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj140 testdata.545011 2024-02-05T06:49:52.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:52.561 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj141 testdata.545011 2024-02-05T06:49:52.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:52.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj142 testdata.545011 2024-02-05T06:49:52.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:52.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj143 testdata.545011 2024-02-05T06:49:53.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:53.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj144 testdata.545011 2024-02-05T06:49:53.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:53.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj145 testdata.545011 2024-02-05T06:49:53.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:53.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj146 testdata.545011 2024-02-05T06:49:53.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:53.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj147 testdata.545011 2024-02-05T06:49:53.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:53.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj148 testdata.545011 2024-02-05T06:49:53.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:53.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj149 testdata.545011 2024-02-05T06:49:54.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:54.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj150 testdata.545011 2024-02-05T06:49:54.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:54.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj151 testdata.545011 2024-02-05T06:49:54.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:54.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj152 testdata.545011 2024-02-05T06:49:54.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:54.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj153 testdata.545011 2024-02-05T06:49:54.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:54.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj154 testdata.545011 2024-02-05T06:49:54.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:54.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj155 testdata.545011 2024-02-05T06:49:55.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:55.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj156 testdata.545011 2024-02-05T06:49:55.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:55.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj157 testdata.545011 2024-02-05T06:49:55.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:55.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj158 testdata.545011 2024-02-05T06:49:55.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:55.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj159 testdata.545011 2024-02-05T06:49:55.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:55.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj160 testdata.545011 2024-02-05T06:49:55.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:55.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj161 testdata.545011 2024-02-05T06:49:56.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:56.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj162 testdata.545011 2024-02-05T06:49:56.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:56.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj163 testdata.545011 2024-02-05T06:49:56.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:56.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj164 testdata.545011 2024-02-05T06:49:56.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:56.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj165 testdata.545011 2024-02-05T06:49:56.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:56.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj166 testdata.545011 2024-02-05T06:49:56.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:56.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj167 testdata.545011 2024-02-05T06:49:57.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:57.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj168 testdata.545011 2024-02-05T06:49:57.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:57.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj169 testdata.545011 2024-02-05T06:49:57.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:57.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj170 testdata.545011 2024-02-05T06:49:57.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:57.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj171 testdata.545011 2024-02-05T06:49:57.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:57.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj172 testdata.545011 2024-02-05T06:49:57.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:57.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj173 testdata.545011 2024-02-05T06:49:58.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:58.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj174 testdata.545011 2024-02-05T06:49:58.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:58.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj175 testdata.545011 2024-02-05T06:49:58.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:58.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj176 testdata.545011 2024-02-05T06:49:58.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:58.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj177 testdata.545011 2024-02-05T06:49:58.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:58.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj178 testdata.545011 2024-02-05T06:49:58.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:58.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj179 testdata.545011 2024-02-05T06:49:59.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:59.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj180 testdata.545011 2024-02-05T06:49:59.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:59.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj181 testdata.545011 2024-02-05T06:49:59.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:59.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj182 testdata.545011 2024-02-05T06:49:59.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:59.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj183 testdata.545011 2024-02-05T06:49:59.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:59.714 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj184 testdata.545011 2024-02-05T06:49:59.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:49:59.889 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj185 testdata.545011 2024-02-05T06:50:00.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:00.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj186 testdata.545011 2024-02-05T06:50:00.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:00.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj187 testdata.545011 2024-02-05T06:50:00.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:00.358 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj188 testdata.545011 2024-02-05T06:50:00.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:00.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj189 testdata.545011 2024-02-05T06:50:00.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:00.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj190 testdata.545011 2024-02-05T06:50:00.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:00.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj191 testdata.545011 2024-02-05T06:50:01.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:01.036 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj192 testdata.545011 2024-02-05T06:50:01.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:01.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj193 testdata.545011 2024-02-05T06:50:01.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:01.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj194 testdata.545011 2024-02-05T06:50:01.513 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:01.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj195 testdata.545011 2024-02-05T06:50:01.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:01.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj196 testdata.545011 2024-02-05T06:50:01.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:01.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj197 testdata.545011 2024-02-05T06:50:02.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:02.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj198 testdata.545011 2024-02-05T06:50:02.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:02.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj199 testdata.545011 2024-02-05T06:50:02.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:02.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj200 testdata.545011 2024-02-05T06:50:02.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:02.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj201 testdata.545011 2024-02-05T06:50:02.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:02.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj202 testdata.545011 2024-02-05T06:50:02.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:02.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj203 testdata.545011 2024-02-05T06:50:03.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:03.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj204 testdata.545011 2024-02-05T06:50:03.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:03.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj205 testdata.545011 2024-02-05T06:50:03.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:03.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj206 testdata.545011 2024-02-05T06:50:03.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:03.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj207 testdata.545011 2024-02-05T06:50:03.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:03.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj208 testdata.545011 2024-02-05T06:50:03.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:03.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj209 testdata.545011 2024-02-05T06:50:03.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:03.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj210 testdata.545011 2024-02-05T06:50:04.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:04.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj211 testdata.545011 2024-02-05T06:50:04.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:04.277 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj212 testdata.545011 2024-02-05T06:50:04.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:04.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj213 testdata.545011 2024-02-05T06:50:04.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:04.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj214 testdata.545011 2024-02-05T06:50:04.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:04.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj215 testdata.545011 2024-02-05T06:50:05.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:05.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj216 testdata.545011 2024-02-05T06:50:05.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:05.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj217 testdata.545011 2024-02-05T06:50:05.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:05.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj218 testdata.545011 2024-02-05T06:50:05.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:05.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj219 testdata.545011 2024-02-05T06:50:05.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:05.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj220 testdata.545011 2024-02-05T06:50:05.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:05.832 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj221 testdata.545011 2024-02-05T06:50:05.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:05.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj222 testdata.545011 2024-02-05T06:50:06.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:06.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj223 testdata.545011 2024-02-05T06:50:06.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:06.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj224 testdata.545011 2024-02-05T06:50:06.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:06.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj225 testdata.545011 2024-02-05T06:50:06.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:06.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj226 testdata.545011 2024-02-05T06:50:06.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:06.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj227 testdata.545011 2024-02-05T06:50:06.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:06.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj228 testdata.545011 2024-02-05T06:50:07.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:07.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj229 testdata.545011 2024-02-05T06:50:07.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:07.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj230 testdata.545011 2024-02-05T06:50:07.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:07.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj231 testdata.545011 2024-02-05T06:50:07.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:07.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj232 testdata.545011 2024-02-05T06:50:07.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:07.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj233 testdata.545011 2024-02-05T06:50:07.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:07.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj234 testdata.545011 2024-02-05T06:50:08.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:08.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj235 testdata.545011 2024-02-05T06:50:08.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:08.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj236 testdata.545011 2024-02-05T06:50:08.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:08.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj237 testdata.545011 2024-02-05T06:50:08.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:08.626 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj238 testdata.545011 2024-02-05T06:50:08.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:08.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj239 testdata.545011 2024-02-05T06:50:08.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:08.943 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj240 testdata.545011 2024-02-05T06:50:09.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:09.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj241 testdata.545011 2024-02-05T06:50:09.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:09.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj242 testdata.545011 2024-02-05T06:50:09.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:09.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj243 testdata.545011 2024-02-05T06:50:09.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:09.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj244 testdata.545011 2024-02-05T06:50:09.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:09.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj245 testdata.545011 2024-02-05T06:50:09.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:09.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj246 testdata.545011 2024-02-05T06:50:10.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:10.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj247 testdata.545011 2024-02-05T06:50:10.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:10.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj248 testdata.545011 2024-02-05T06:50:10.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:10.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj249 testdata.545011 2024-02-05T06:50:10.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:10.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj250 testdata.545011 2024-02-05T06:50:10.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:10.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj251 testdata.545011 2024-02-05T06:50:10.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:10.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj252 testdata.545011 2024-02-05T06:50:11.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:11.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj253 testdata.545011 2024-02-05T06:50:11.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:11.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj254 testdata.545011 2024-02-05T06:50:11.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:11.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj255 testdata.545011 2024-02-05T06:50:11.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:11.556 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj256 testdata.545011 2024-02-05T06:50:11.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:11.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj257 testdata.545011 2024-02-05T06:50:11.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:11.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj258 testdata.545011 2024-02-05T06:50:12.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:12.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj259 testdata.545011 2024-02-05T06:50:12.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:12.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj260 testdata.545011 2024-02-05T06:50:12.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:12.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj261 testdata.545011 2024-02-05T06:50:12.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:12.570 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj262 testdata.545011 2024-02-05T06:50:12.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:12.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj263 testdata.545011 2024-02-05T06:50:12.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:12.929 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj264 testdata.545011 2024-02-05T06:50:13.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:13.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj265 testdata.545011 2024-02-05T06:50:13.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:13.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj266 testdata.545011 2024-02-05T06:50:13.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:13.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj267 testdata.545011 2024-02-05T06:50:13.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:13.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj268 testdata.545011 2024-02-05T06:50:13.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:13.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj269 testdata.545011 2024-02-05T06:50:13.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:13.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj270 testdata.545011 2024-02-05T06:50:13.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:13.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj271 testdata.545011 2024-02-05T06:50:14.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:14.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj272 testdata.545011 2024-02-05T06:50:14.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:14.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj273 testdata.545011 2024-02-05T06:50:14.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:14.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj274 testdata.545011 2024-02-05T06:50:14.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:14.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj275 testdata.545011 2024-02-05T06:50:14.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:14.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj276 testdata.545011 2024-02-05T06:50:15.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:15.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj277 testdata.545011 2024-02-05T06:50:15.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:15.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj278 testdata.545011 2024-02-05T06:50:15.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:15.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj279 testdata.545011 2024-02-05T06:50:15.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:15.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj280 testdata.545011 2024-02-05T06:50:15.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:15.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj281 testdata.545011 2024-02-05T06:50:15.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:15.880 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj282 testdata.545011 2024-02-05T06:50:16.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:16.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj283 testdata.545011 2024-02-05T06:50:16.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:16.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj284 testdata.545011 2024-02-05T06:50:16.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:16.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj285 testdata.545011 2024-02-05T06:50:16.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:16.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj286 testdata.545011 2024-02-05T06:50:16.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:16.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj287 testdata.545011 2024-02-05T06:50:16.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:16.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj288 testdata.545011 2024-02-05T06:50:17.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:17.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj289 testdata.545011 2024-02-05T06:50:17.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:17.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj290 testdata.545011 2024-02-05T06:50:17.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:17.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj291 testdata.545011 2024-02-05T06:50:17.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:17.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj292 testdata.545011 2024-02-05T06:50:17.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:17.720 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj293 testdata.545011 2024-02-05T06:50:17.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:17.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj294 testdata.545011 2024-02-05T06:50:18.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:18.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj295 testdata.545011 2024-02-05T06:50:18.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:18.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj296 testdata.545011 2024-02-05T06:50:18.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:18.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj297 testdata.545011 2024-02-05T06:50:18.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:18.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj298 testdata.545011 2024-02-05T06:50:18.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:18.718 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj299 testdata.545011 2024-02-05T06:50:18.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:18.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj300 testdata.545011 2024-02-05T06:50:19.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:19.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj301 testdata.545011 2024-02-05T06:50:19.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:19.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj302 testdata.545011 2024-02-05T06:50:19.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:19.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj303 testdata.545011 2024-02-05T06:50:19.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:19.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj304 testdata.545011 2024-02-05T06:50:19.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:19.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj305 testdata.545011 2024-02-05T06:50:19.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:19.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj306 testdata.545011 2024-02-05T06:50:20.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:20.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj307 testdata.545011 2024-02-05T06:50:20.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:20.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj308 testdata.545011 2024-02-05T06:50:20.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:20.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj309 testdata.545011 2024-02-05T06:50:20.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:20.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj310 testdata.545011 2024-02-05T06:50:20.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:20.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj311 testdata.545011 2024-02-05T06:50:21.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:21.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj312 testdata.545011 2024-02-05T06:50:21.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:21.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj313 testdata.545011 2024-02-05T06:50:21.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:21.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj314 testdata.545011 2024-02-05T06:50:21.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:21.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj315 testdata.545011 2024-02-05T06:50:21.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:21.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj316 testdata.545011 2024-02-05T06:50:21.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:21.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj317 testdata.545011 2024-02-05T06:50:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:22.167 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj318 testdata.545011 2024-02-05T06:50:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:22.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj319 testdata.545011 2024-02-05T06:50:22.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:22.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj320 testdata.545011 2024-02-05T06:50:22.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:22.712 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj321 testdata.545011 2024-02-05T06:50:22.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:22.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj322 testdata.545011 2024-02-05T06:50:23.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:23.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj323 testdata.545011 2024-02-05T06:50:23.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:23.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj324 testdata.545011 2024-02-05T06:50:23.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:23.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj325 testdata.545011 2024-02-05T06:50:23.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:23.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj326 testdata.545011 2024-02-05T06:50:23.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:23.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj327 testdata.545011 2024-02-05T06:50:23.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:23.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj328 testdata.545011 2024-02-05T06:50:23.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:23.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj329 testdata.545011 2024-02-05T06:50:24.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:24.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj330 testdata.545011 2024-02-05T06:50:24.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:24.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj331 testdata.545011 2024-02-05T06:50:24.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:24.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj332 testdata.545011 2024-02-05T06:50:24.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:24.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj333 testdata.545011 2024-02-05T06:50:24.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:24.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj334 testdata.545011 2024-02-05T06:50:24.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:24.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj335 testdata.545011 2024-02-05T06:50:25.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:25.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj336 testdata.545011 2024-02-05T06:50:25.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:25.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj337 testdata.545011 2024-02-05T06:50:25.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:25.408 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj338 testdata.545011 2024-02-05T06:50:25.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:25.576 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj339 testdata.545011 2024-02-05T06:50:25.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:25.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj340 testdata.545011 2024-02-05T06:50:25.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:25.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj341 testdata.545011 2024-02-05T06:50:26.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:26.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj342 testdata.545011 2024-02-05T06:50:26.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:26.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj343 testdata.545011 2024-02-05T06:50:26.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:26.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj344 testdata.545011 2024-02-05T06:50:26.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:26.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj345 testdata.545011 2024-02-05T06:50:26.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:26.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj346 testdata.545011 2024-02-05T06:50:26.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:26.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj347 testdata.545011 2024-02-05T06:50:27.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:27.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj348 testdata.545011 2024-02-05T06:50:27.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:27.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj349 testdata.545011 2024-02-05T06:50:27.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:27.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj350 testdata.545011 2024-02-05T06:50:27.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:27.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj351 testdata.545011 2024-02-05T06:50:27.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:27.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj352 testdata.545011 2024-02-05T06:50:27.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:27.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj353 testdata.545011 2024-02-05T06:50:28.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:28.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj354 testdata.545011 2024-02-05T06:50:28.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:28.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj355 testdata.545011 2024-02-05T06:50:28.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:28.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj356 testdata.545011 2024-02-05T06:50:28.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:28.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj357 testdata.545011 2024-02-05T06:50:28.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:28.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj358 testdata.545011 2024-02-05T06:50:28.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:28.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj359 testdata.545011 2024-02-05T06:50:29.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:29.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj360 testdata.545011 2024-02-05T06:50:29.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:29.263 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj361 testdata.545011 2024-02-05T06:50:29.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:29.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj362 testdata.545011 2024-02-05T06:50:29.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:29.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj363 testdata.545011 2024-02-05T06:50:29.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:29.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj364 testdata.545011 2024-02-05T06:50:29.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:29.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj365 testdata.545011 2024-02-05T06:50:30.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:30.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj366 testdata.545011 2024-02-05T06:50:30.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:30.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj367 testdata.545011 2024-02-05T06:50:30.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:30.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj368 testdata.545011 2024-02-05T06:50:30.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:30.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj369 testdata.545011 2024-02-05T06:50:30.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:30.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj370 testdata.545011 2024-02-05T06:50:30.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:30.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj371 testdata.545011 2024-02-05T06:50:31.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:31.063 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj372 testdata.545011 2024-02-05T06:50:31.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:31.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj373 testdata.545011 2024-02-05T06:50:31.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:31.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj374 testdata.545011 2024-02-05T06:50:31.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:31.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj375 testdata.545011 2024-02-05T06:50:31.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:31.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj376 testdata.545011 2024-02-05T06:50:31.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:31.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj377 testdata.545011 2024-02-05T06:50:31.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:31.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj378 testdata.545011 2024-02-05T06:50:32.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:32.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj379 testdata.545011 2024-02-05T06:50:32.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:32.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj380 testdata.545011 2024-02-05T06:50:32.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:32.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj381 testdata.545011 2024-02-05T06:50:32.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:32.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj382 testdata.545011 2024-02-05T06:50:32.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:32.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj383 testdata.545011 2024-02-05T06:50:32.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:32.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj384 testdata.545011 2024-02-05T06:50:33.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:33.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj385 testdata.545011 2024-02-05T06:50:33.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:33.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj386 testdata.545011 2024-02-05T06:50:33.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:33.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj387 testdata.545011 2024-02-05T06:50:33.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:33.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj388 testdata.545011 2024-02-05T06:50:33.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:33.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj389 testdata.545011 2024-02-05T06:50:33.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:33.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj390 testdata.545011 2024-02-05T06:50:34.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:34.146 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj391 testdata.545011 2024-02-05T06:50:34.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:34.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj392 testdata.545011 2024-02-05T06:50:34.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:34.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj393 testdata.545011 2024-02-05T06:50:34.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:34.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj394 testdata.545011 2024-02-05T06:50:34.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:34.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj395 testdata.545011 2024-02-05T06:50:34.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:34.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj396 testdata.545011 2024-02-05T06:50:35.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:35.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj397 testdata.545011 2024-02-05T06:50:35.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:35.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj398 testdata.545011 2024-02-05T06:50:35.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:35.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj399 testdata.545011 2024-02-05T06:50:35.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:35.645 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj400 testdata.545011 2024-02-05T06:50:35.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:35.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj401 testdata.545011 2024-02-05T06:50:36.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:36.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj402 testdata.545011 2024-02-05T06:50:36.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:36.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj403 testdata.545011 2024-02-05T06:50:36.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:36.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj404 testdata.545011 2024-02-05T06:50:36.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:36.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj405 testdata.545011 2024-02-05T06:50:36.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:36.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj406 testdata.545011 2024-02-05T06:50:36.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:36.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj407 testdata.545011 2024-02-05T06:50:36.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:36.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj408 testdata.545011 2024-02-05T06:50:37.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:37.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj409 testdata.545011 2024-02-05T06:50:37.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:37.312 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj410 testdata.545011 2024-02-05T06:50:37.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:37.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj411 testdata.545011 2024-02-05T06:50:37.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:37.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj412 testdata.545011 2024-02-05T06:50:37.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:37.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj413 testdata.545011 2024-02-05T06:50:37.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:37.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj414 testdata.545011 2024-02-05T06:50:38.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:38.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj415 testdata.545011 2024-02-05T06:50:38.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:38.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj416 testdata.545011 2024-02-05T06:50:38.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:38.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj417 testdata.545011 2024-02-05T06:50:38.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:38.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj418 testdata.545011 2024-02-05T06:50:38.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:38.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj419 testdata.545011 2024-02-05T06:50:38.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:38.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj420 testdata.545011 2024-02-05T06:50:39.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:39.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj421 testdata.545011 2024-02-05T06:50:39.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:39.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj422 testdata.545011 2024-02-05T06:50:39.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:39.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj423 testdata.545011 2024-02-05T06:50:39.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:39.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj424 testdata.545011 2024-02-05T06:50:39.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:39.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj425 testdata.545011 2024-02-05T06:50:39.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:39.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj426 testdata.545011 2024-02-05T06:50:40.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:40.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj427 testdata.545011 2024-02-05T06:50:40.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:40.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj428 testdata.545011 2024-02-05T06:50:40.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:40.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj429 testdata.545011 2024-02-05T06:50:40.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:40.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj430 testdata.545011 2024-02-05T06:50:40.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:40.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj431 testdata.545011 2024-02-05T06:50:41.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:41.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj432 testdata.545011 2024-02-05T06:50:41.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:41.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj433 testdata.545011 2024-02-05T06:50:41.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:41.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj434 testdata.545011 2024-02-05T06:50:41.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:41.449 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj435 testdata.545011 2024-02-05T06:50:41.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:41.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj436 testdata.545011 2024-02-05T06:50:41.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:41.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj437 testdata.545011 2024-02-05T06:50:41.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:41.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj438 testdata.545011 2024-02-05T06:50:42.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:42.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj439 testdata.545011 2024-02-05T06:50:42.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:42.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj440 testdata.545011 2024-02-05T06:50:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:42.411 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj441 testdata.545011 2024-02-05T06:50:42.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:42.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj442 testdata.545011 2024-02-05T06:50:42.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:42.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj443 testdata.545011 2024-02-05T06:50:42.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:42.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj444 testdata.545011 2024-02-05T06:50:43.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:43.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj445 testdata.545011 2024-02-05T06:50:43.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:43.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj446 testdata.545011 2024-02-05T06:50:43.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:43.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj447 testdata.545011 2024-02-05T06:50:43.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:43.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj448 testdata.545011 2024-02-05T06:50:43.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:43.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj449 testdata.545011 2024-02-05T06:50:43.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:43.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj450 testdata.545011 2024-02-05T06:50:44.053 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:44.054 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj451 testdata.545011 2024-02-05T06:50:44.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:44.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj452 testdata.545011 2024-02-05T06:50:44.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:44.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj453 testdata.545011 2024-02-05T06:50:44.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:44.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj454 testdata.545011 2024-02-05T06:50:44.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:44.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj455 testdata.545011 2024-02-05T06:50:44.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:44.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj456 testdata.545011 2024-02-05T06:50:45.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:45.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj457 testdata.545011 2024-02-05T06:50:45.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:45.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj458 testdata.545011 2024-02-05T06:50:45.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:45.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj459 testdata.545011 2024-02-05T06:50:45.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:45.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj460 testdata.545011 2024-02-05T06:50:45.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:45.670 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj461 testdata.545011 2024-02-05T06:50:45.855 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:45.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj462 testdata.545011 2024-02-05T06:50:45.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:45.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj463 testdata.545011 2024-02-05T06:50:46.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:46.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj464 testdata.545011 2024-02-05T06:50:46.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:46.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj465 testdata.545011 2024-02-05T06:50:46.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:46.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj466 testdata.545011 2024-02-05T06:50:46.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:46.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj467 testdata.545011 2024-02-05T06:50:46.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:46.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj468 testdata.545011 2024-02-05T06:50:46.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:46.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj469 testdata.545011 2024-02-05T06:50:47.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:47.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj470 testdata.545011 2024-02-05T06:50:47.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:47.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj471 testdata.545011 2024-02-05T06:50:47.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:47.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj472 testdata.545011 2024-02-05T06:50:47.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:47.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj473 testdata.545011 2024-02-05T06:50:47.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:47.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj474 testdata.545011 2024-02-05T06:50:47.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:47.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj475 testdata.545011 2024-02-05T06:50:48.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:48.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj476 testdata.545011 2024-02-05T06:50:48.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:48.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj477 testdata.545011 2024-02-05T06:50:48.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:48.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj478 testdata.545011 2024-02-05T06:50:48.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:48.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj479 testdata.545011 2024-02-05T06:50:48.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:48.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj480 testdata.545011 2024-02-05T06:50:48.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:48.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj481 testdata.545011 2024-02-05T06:50:49.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:49.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj482 testdata.545011 2024-02-05T06:50:49.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:49.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj483 testdata.545011 2024-02-05T06:50:49.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:49.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj484 testdata.545011 2024-02-05T06:50:49.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:49.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj485 testdata.545011 2024-02-05T06:50:49.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:49.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj486 testdata.545011 2024-02-05T06:50:49.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:49.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj487 testdata.545011 2024-02-05T06:50:49.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:49.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj488 testdata.545011 2024-02-05T06:50:50.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:50.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj489 testdata.545011 2024-02-05T06:50:50.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:50.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj490 testdata.545011 2024-02-05T06:50:50.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:50.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj491 testdata.545011 2024-02-05T06:50:50.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:50.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj492 testdata.545011 2024-02-05T06:50:50.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:50.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj493 testdata.545011 2024-02-05T06:50:51.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:51.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj494 testdata.545011 2024-02-05T06:50:51.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:51.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj495 testdata.545011 2024-02-05T06:50:51.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:51.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj496 testdata.545011 2024-02-05T06:50:51.783 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:51.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj497 testdata.545011 2024-02-05T06:50:51.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:51.961 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj498 testdata.545011 2024-02-05T06:50:52.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:52.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj499 testdata.545011 2024-02-05T06:50:52.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:52.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj500 testdata.545011 2024-02-05T06:50:52.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:52.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj501 testdata.545011 2024-02-05T06:50:52.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:52.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj502 testdata.545011 2024-02-05T06:50:52.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:52.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj503 testdata.545011 2024-02-05T06:50:53.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:53.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj504 testdata.545011 2024-02-05T06:50:53.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:53.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj505 testdata.545011 2024-02-05T06:50:53.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:53.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj506 testdata.545011 2024-02-05T06:50:53.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:53.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj507 testdata.545011 2024-02-05T06:50:53.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:53.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj508 testdata.545011 2024-02-05T06:50:53.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:53.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj509 testdata.545011 2024-02-05T06:50:53.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:53.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj510 testdata.545011 2024-02-05T06:50:54.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:54.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj511 testdata.545011 2024-02-05T06:50:54.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:54.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj512 testdata.545011 2024-02-05T06:50:54.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:54.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj513 testdata.545011 2024-02-05T06:50:54.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:54.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj514 testdata.545011 2024-02-05T06:50:54.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:54.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj515 testdata.545011 2024-02-05T06:50:54.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:54.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj516 testdata.545011 2024-02-05T06:50:55.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:55.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj517 testdata.545011 2024-02-05T06:50:55.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:55.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj518 testdata.545011 2024-02-05T06:50:55.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:55.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj519 testdata.545011 2024-02-05T06:50:55.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:55.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj520 testdata.545011 2024-02-05T06:50:55.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:55.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj521 testdata.545011 2024-02-05T06:50:55.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:55.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj522 testdata.545011 2024-02-05T06:50:56.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:56.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj523 testdata.545011 2024-02-05T06:50:56.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:56.231 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj524 testdata.545011 2024-02-05T06:50:56.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:56.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj525 testdata.545011 2024-02-05T06:50:56.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:56.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj526 testdata.545011 2024-02-05T06:50:56.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:56.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj527 testdata.545011 2024-02-05T06:50:56.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:56.876 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj528 testdata.545011 2024-02-05T06:50:57.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:57.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj529 testdata.545011 2024-02-05T06:50:57.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:57.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj530 testdata.545011 2024-02-05T06:50:57.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:57.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj531 testdata.545011 2024-02-05T06:50:57.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:57.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj532 testdata.545011 2024-02-05T06:50:57.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:57.722 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj533 testdata.545011 2024-02-05T06:50:57.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:57.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj534 testdata.545011 2024-02-05T06:50:58.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:58.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj535 testdata.545011 2024-02-05T06:50:58.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:58.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj536 testdata.545011 2024-02-05T06:50:58.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:58.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj537 testdata.545011 2024-02-05T06:50:58.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:58.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj538 testdata.545011 2024-02-05T06:50:58.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:58.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj539 testdata.545011 2024-02-05T06:50:58.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:58.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj540 testdata.545011 2024-02-05T06:50:59.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:59.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj541 testdata.545011 2024-02-05T06:50:59.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:59.230 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj542 testdata.545011 2024-02-05T06:50:59.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:59.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj543 testdata.545011 2024-02-05T06:50:59.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:59.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj544 testdata.545011 2024-02-05T06:50:59.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:59.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj545 testdata.545011 2024-02-05T06:50:59.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:59.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj546 testdata.545011 2024-02-05T06:50:59.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:50:59.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj547 testdata.545011 2024-02-05T06:51:00.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:00.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj548 testdata.545011 2024-02-05T06:51:00.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:00.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj549 testdata.545011 2024-02-05T06:51:00.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:00.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj550 testdata.545011 2024-02-05T06:51:00.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:00.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj551 testdata.545011 2024-02-05T06:51:00.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:00.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj552 testdata.545011 2024-02-05T06:51:01.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:01.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj553 testdata.545011 2024-02-05T06:51:01.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:01.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj554 testdata.545011 2024-02-05T06:51:01.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:01.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj555 testdata.545011 2024-02-05T06:51:01.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:01.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj556 testdata.545011 2024-02-05T06:51:01.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:01.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj557 testdata.545011 2024-02-05T06:51:01.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:01.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj558 testdata.545011 2024-02-05T06:51:01.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:01.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj559 testdata.545011 2024-02-05T06:51:02.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:02.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj560 testdata.545011 2024-02-05T06:51:02.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:02.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj561 testdata.545011 2024-02-05T06:51:02.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:02.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj562 testdata.545011 2024-02-05T06:51:02.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:02.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj563 testdata.545011 2024-02-05T06:51:02.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:02.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj564 testdata.545011 2024-02-05T06:51:02.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:02.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj565 testdata.545011 2024-02-05T06:51:03.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:03.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj566 testdata.545011 2024-02-05T06:51:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:03.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj567 testdata.545011 2024-02-05T06:51:03.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:03.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj568 testdata.545011 2024-02-05T06:51:03.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:03.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj569 testdata.545011 2024-02-05T06:51:03.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:03.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj570 testdata.545011 2024-02-05T06:51:03.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:03.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj571 testdata.545011 2024-02-05T06:51:04.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:04.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj572 testdata.545011 2024-02-05T06:51:04.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:04.238 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj573 testdata.545011 2024-02-05T06:51:04.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:04.389 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj574 testdata.545011 2024-02-05T06:51:04.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:04.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj575 testdata.545011 2024-02-05T06:51:04.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:04.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj576 testdata.545011 2024-02-05T06:51:04.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:04.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj577 testdata.545011 2024-02-05T06:51:05.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:05.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj578 testdata.545011 2024-02-05T06:51:05.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:05.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj579 testdata.545011 2024-02-05T06:51:05.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:05.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj580 testdata.545011 2024-02-05T06:51:05.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:05.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj581 testdata.545011 2024-02-05T06:51:05.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:05.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj582 testdata.545011 2024-02-05T06:51:05.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:05.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj583 testdata.545011 2024-02-05T06:51:06.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:06.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj584 testdata.545011 2024-02-05T06:51:06.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:06.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj585 testdata.545011 2024-02-05T06:51:06.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:06.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj586 testdata.545011 2024-02-05T06:51:06.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:06.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj587 testdata.545011 2024-02-05T06:51:06.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:06.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj588 testdata.545011 2024-02-05T06:51:06.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:06.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj589 testdata.545011 2024-02-05T06:51:07.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:07.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj590 testdata.545011 2024-02-05T06:51:07.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:07.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj591 testdata.545011 2024-02-05T06:51:07.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:07.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj592 testdata.545011 2024-02-05T06:51:07.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:07.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj593 testdata.545011 2024-02-05T06:51:07.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:07.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj594 testdata.545011 2024-02-05T06:51:07.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:07.800 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj595 testdata.545011 2024-02-05T06:51:07.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:07.960 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj596 testdata.545011 2024-02-05T06:51:08.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:08.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj597 testdata.545011 2024-02-05T06:51:08.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:08.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj598 testdata.545011 2024-02-05T06:51:08.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:08.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj599 testdata.545011 2024-02-05T06:51:08.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:08.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj600 testdata.545011 2024-02-05T06:51:08.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:08.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj601 testdata.545011 2024-02-05T06:51:08.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:08.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj602 testdata.545011 2024-02-05T06:51:09.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:09.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj603 testdata.545011 2024-02-05T06:51:09.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:09.241 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj604 testdata.545011 2024-02-05T06:51:09.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:09.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj605 testdata.545011 2024-02-05T06:51:09.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:09.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj606 testdata.545011 2024-02-05T06:51:09.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:09.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj607 testdata.545011 2024-02-05T06:51:09.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:09.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj608 testdata.545011 2024-02-05T06:51:10.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:10.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj609 testdata.545011 2024-02-05T06:51:10.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:10.213 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj610 testdata.545011 2024-02-05T06:51:10.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:10.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj611 testdata.545011 2024-02-05T06:51:10.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:10.514 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj612 testdata.545011 2024-02-05T06:51:10.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:10.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj613 testdata.545011 2024-02-05T06:51:10.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:10.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj614 testdata.545011 2024-02-05T06:51:10.992 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:10.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj615 testdata.545011 2024-02-05T06:51:11.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:11.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj616 testdata.545011 2024-02-05T06:51:11.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:11.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj617 testdata.545011 2024-02-05T06:51:11.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:11.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj618 testdata.545011 2024-02-05T06:51:11.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:11.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj619 testdata.545011 2024-02-05T06:51:11.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:11.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj620 testdata.545011 2024-02-05T06:51:12.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:12.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj621 testdata.545011 2024-02-05T06:51:12.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:12.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj622 testdata.545011 2024-02-05T06:51:12.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:12.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj623 testdata.545011 2024-02-05T06:51:12.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:12.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj624 testdata.545011 2024-02-05T06:51:12.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:12.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj625 testdata.545011 2024-02-05T06:51:12.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:12.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj626 testdata.545011 2024-02-05T06:51:13.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:13.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj627 testdata.545011 2024-02-05T06:51:13.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:13.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj628 testdata.545011 2024-02-05T06:51:13.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:13.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj629 testdata.545011 2024-02-05T06:51:13.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:13.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj630 testdata.545011 2024-02-05T06:51:13.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:13.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj631 testdata.545011 2024-02-05T06:51:13.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:13.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj632 testdata.545011 2024-02-05T06:51:14.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:14.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj633 testdata.545011 2024-02-05T06:51:14.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:14.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj634 testdata.545011 2024-02-05T06:51:14.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:14.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj635 testdata.545011 2024-02-05T06:51:14.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:14.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj636 testdata.545011 2024-02-05T06:51:14.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:14.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj637 testdata.545011 2024-02-05T06:51:14.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:14.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj638 testdata.545011 2024-02-05T06:51:15.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:15.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj639 testdata.545011 2024-02-05T06:51:15.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:15.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj640 testdata.545011 2024-02-05T06:51:15.322 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:15.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj641 testdata.545011 2024-02-05T06:51:15.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:15.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj642 testdata.545011 2024-02-05T06:51:15.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:15.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj643 testdata.545011 2024-02-05T06:51:15.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:15.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj644 testdata.545011 2024-02-05T06:51:15.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:15.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj645 testdata.545011 2024-02-05T06:51:16.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:16.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj646 testdata.545011 2024-02-05T06:51:16.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:16.320 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj647 testdata.545011 2024-02-05T06:51:16.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:16.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj648 testdata.545011 2024-02-05T06:51:16.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:16.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj649 testdata.545011 2024-02-05T06:51:16.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:16.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj650 testdata.545011 2024-02-05T06:51:16.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:16.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj651 testdata.545011 2024-02-05T06:51:17.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:17.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj652 testdata.545011 2024-02-05T06:51:17.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:17.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj653 testdata.545011 2024-02-05T06:51:17.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:17.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj654 testdata.545011 2024-02-05T06:51:17.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:17.635 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj655 testdata.545011 2024-02-05T06:51:17.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:17.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj656 testdata.545011 2024-02-05T06:51:17.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:17.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj657 testdata.545011 2024-02-05T06:51:18.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:18.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj658 testdata.545011 2024-02-05T06:51:18.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:18.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj659 testdata.545011 2024-02-05T06:51:18.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:18.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj660 testdata.545011 2024-02-05T06:51:18.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:18.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj661 testdata.545011 2024-02-05T06:51:18.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:18.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj662 testdata.545011 2024-02-05T06:51:18.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:18.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj663 testdata.545011 2024-02-05T06:51:19.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:19.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj664 testdata.545011 2024-02-05T06:51:19.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:19.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj665 testdata.545011 2024-02-05T06:51:19.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:19.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj666 testdata.545011 2024-02-05T06:51:19.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:19.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj667 testdata.545011 2024-02-05T06:51:19.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:19.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj668 testdata.545011 2024-02-05T06:51:20.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:20.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj669 testdata.545011 2024-02-05T06:51:20.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:20.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj670 testdata.545011 2024-02-05T06:51:20.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:20.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj671 testdata.545011 2024-02-05T06:51:20.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:20.451 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj672 testdata.545011 2024-02-05T06:51:20.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:20.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj673 testdata.545011 2024-02-05T06:51:20.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:20.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj674 testdata.545011 2024-02-05T06:51:20.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:20.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj675 testdata.545011 2024-02-05T06:51:21.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:21.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj676 testdata.545011 2024-02-05T06:51:21.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:21.281 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj677 testdata.545011 2024-02-05T06:51:21.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:21.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj678 testdata.545011 2024-02-05T06:51:21.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:21.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj679 testdata.545011 2024-02-05T06:51:21.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:21.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj680 testdata.545011 2024-02-05T06:51:21.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:21.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj681 testdata.545011 2024-02-05T06:51:22.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:22.171 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj682 testdata.545011 2024-02-05T06:51:22.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:22.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj683 testdata.545011 2024-02-05T06:51:22.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:22.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj684 testdata.545011 2024-02-05T06:51:22.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:22.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj685 testdata.545011 2024-02-05T06:51:22.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:22.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj686 testdata.545011 2024-02-05T06:51:23.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:23.090 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj687 testdata.545011 2024-02-05T06:51:23.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:23.240 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj688 testdata.545011 2024-02-05T06:51:23.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:23.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj689 testdata.545011 2024-02-05T06:51:23.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:23.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj690 testdata.545011 2024-02-05T06:51:23.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:23.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj691 testdata.545011 2024-02-05T06:51:23.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:23.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj692 testdata.545011 2024-02-05T06:51:24.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:24.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj693 testdata.545011 2024-02-05T06:51:24.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:24.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj694 testdata.545011 2024-02-05T06:51:24.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:24.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj695 testdata.545011 2024-02-05T06:51:24.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:24.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj696 testdata.545011 2024-02-05T06:51:24.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:24.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj697 testdata.545011 2024-02-05T06:51:24.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:24.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj698 testdata.545011 2024-02-05T06:51:25.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:25.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj699 testdata.545011 2024-02-05T06:51:25.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:25.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj700 testdata.545011 2024-02-05T06:51:25.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:25.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj701 testdata.545011 2024-02-05T06:51:25.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:25.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj702 testdata.545011 2024-02-05T06:51:25.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:25.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj703 testdata.545011 2024-02-05T06:51:25.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:25.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj704 testdata.545011 2024-02-05T06:51:26.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:26.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj705 testdata.545011 2024-02-05T06:51:26.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:26.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj706 testdata.545011 2024-02-05T06:51:26.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:26.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj707 testdata.545011 2024-02-05T06:51:26.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:26.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj708 testdata.545011 2024-02-05T06:51:26.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:26.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj709 testdata.545011 2024-02-05T06:51:26.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:26.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj710 testdata.545011 2024-02-05T06:51:27.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:27.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj711 testdata.545011 2024-02-05T06:51:27.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:27.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj712 testdata.545011 2024-02-05T06:51:27.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:27.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj713 testdata.545011 2024-02-05T06:51:27.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:27.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj714 testdata.545011 2024-02-05T06:51:27.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:27.713 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj715 testdata.545011 2024-02-05T06:51:27.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:27.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj716 testdata.545011 2024-02-05T06:51:28.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:28.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj717 testdata.545011 2024-02-05T06:51:28.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:28.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj718 testdata.545011 2024-02-05T06:51:28.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:28.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj719 testdata.545011 2024-02-05T06:51:28.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:28.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj720 testdata.545011 2024-02-05T06:51:28.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:28.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj721 testdata.545011 2024-02-05T06:51:28.868 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:28.869 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj722 testdata.545011 2024-02-05T06:51:29.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:29.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj723 testdata.545011 2024-02-05T06:51:29.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:29.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj724 testdata.545011 2024-02-05T06:51:29.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:29.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj725 testdata.545011 2024-02-05T06:51:29.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:29.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj726 testdata.545011 2024-02-05T06:51:29.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:29.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj727 testdata.545011 2024-02-05T06:51:29.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:29.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj728 testdata.545011 2024-02-05T06:51:30.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:30.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj729 testdata.545011 2024-02-05T06:51:30.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:30.208 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj730 testdata.545011 2024-02-05T06:51:30.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:30.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj731 testdata.545011 2024-02-05T06:51:30.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:30.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj732 testdata.545011 2024-02-05T06:51:30.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:30.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj733 testdata.545011 2024-02-05T06:51:30.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:30.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj734 testdata.545011 2024-02-05T06:51:31.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:31.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj735 testdata.545011 2024-02-05T06:51:31.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:31.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj736 testdata.545011 2024-02-05T06:51:31.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:31.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj737 testdata.545011 2024-02-05T06:51:31.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:31.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj738 testdata.545011 2024-02-05T06:51:31.691 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:31.692 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj739 testdata.545011 2024-02-05T06:51:31.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:31.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj740 testdata.545011 2024-02-05T06:51:32.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:32.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj741 testdata.545011 2024-02-05T06:51:32.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:32.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj742 testdata.545011 2024-02-05T06:51:32.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:32.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj743 testdata.545011 2024-02-05T06:51:32.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:32.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj744 testdata.545011 2024-02-05T06:51:32.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:32.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj745 testdata.545011 2024-02-05T06:51:32.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:32.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj746 testdata.545011 2024-02-05T06:51:32.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:32.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj747 testdata.545011 2024-02-05T06:51:33.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:33.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj748 testdata.545011 2024-02-05T06:51:33.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:33.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj749 testdata.545011 2024-02-05T06:51:33.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:33.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj750 testdata.545011 2024-02-05T06:51:33.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:33.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj751 testdata.545011 2024-02-05T06:51:33.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:33.795 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj752 testdata.545011 2024-02-05T06:51:33.979 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:33.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj753 testdata.545011 2024-02-05T06:51:34.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:34.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj754 testdata.545011 2024-02-05T06:51:34.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:34.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj755 testdata.545011 2024-02-05T06:51:34.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:34.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj756 testdata.545011 2024-02-05T06:51:34.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:34.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj757 testdata.545011 2024-02-05T06:51:34.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:34.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj758 testdata.545011 2024-02-05T06:51:34.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:34.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj759 testdata.545011 2024-02-05T06:51:35.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:35.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj760 testdata.545011 2024-02-05T06:51:35.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:35.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj761 testdata.545011 2024-02-05T06:51:35.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:35.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj762 testdata.545011 2024-02-05T06:51:35.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:35.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj763 testdata.545011 2024-02-05T06:51:35.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:35.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj764 testdata.545011 2024-02-05T06:51:35.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:35.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj765 testdata.545011 2024-02-05T06:51:36.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:36.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj766 testdata.545011 2024-02-05T06:51:36.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:36.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj767 testdata.545011 2024-02-05T06:51:36.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:36.365 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj768 testdata.545011 2024-02-05T06:51:36.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:36.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj769 testdata.545011 2024-02-05T06:51:36.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:36.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj770 testdata.545011 2024-02-05T06:51:36.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:36.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj771 testdata.545011 2024-02-05T06:51:37.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:37.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj772 testdata.545011 2024-02-05T06:51:37.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:37.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj773 testdata.545011 2024-02-05T06:51:37.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:37.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj774 testdata.545011 2024-02-05T06:51:37.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:37.548 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj775 testdata.545011 2024-02-05T06:51:37.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:37.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj776 testdata.545011 2024-02-05T06:51:37.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:37.991 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj777 testdata.545011 2024-02-05T06:51:38.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:38.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj778 testdata.545011 2024-02-05T06:51:38.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:38.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj779 testdata.545011 2024-02-05T06:51:38.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:38.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj780 testdata.545011 2024-02-05T06:51:38.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:38.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj781 testdata.545011 2024-02-05T06:51:38.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:38.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj782 testdata.545011 2024-02-05T06:51:38.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:38.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj783 testdata.545011 2024-02-05T06:51:39.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:39.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj784 testdata.545011 2024-02-05T06:51:39.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:39.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj785 testdata.545011 2024-02-05T06:51:39.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:39.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj786 testdata.545011 2024-02-05T06:51:39.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:39.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj787 testdata.545011 2024-02-05T06:51:39.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:39.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj788 testdata.545011 2024-02-05T06:51:39.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:39.867 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj789 testdata.545011 2024-02-05T06:51:40.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:40.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj790 testdata.545011 2024-02-05T06:51:40.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:40.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj791 testdata.545011 2024-02-05T06:51:40.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:40.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj792 testdata.545011 2024-02-05T06:51:40.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:40.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj793 testdata.545011 2024-02-05T06:51:40.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:40.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj794 testdata.545011 2024-02-05T06:51:40.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:40.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj795 testdata.545011 2024-02-05T06:51:41.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:41.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj796 testdata.545011 2024-02-05T06:51:41.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:41.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj797 testdata.545011 2024-02-05T06:51:41.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:41.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj798 testdata.545011 2024-02-05T06:51:41.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:41.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj799 testdata.545011 2024-02-05T06:51:41.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:41.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj800 testdata.545011 2024-02-05T06:51:41.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:41.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj801 testdata.545011 2024-02-05T06:51:42.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:42.020 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj802 testdata.545011 2024-02-05T06:51:42.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:42.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj803 testdata.545011 2024-02-05T06:51:42.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:42.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj804 testdata.545011 2024-02-05T06:51:42.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:42.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj805 testdata.545011 2024-02-05T06:51:42.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:42.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj806 testdata.545011 2024-02-05T06:51:42.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:42.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj807 testdata.545011 2024-02-05T06:51:43.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:43.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj808 testdata.545011 2024-02-05T06:51:43.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:43.218 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj809 testdata.545011 2024-02-05T06:51:43.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:43.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj810 testdata.545011 2024-02-05T06:51:43.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:43.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj811 testdata.545011 2024-02-05T06:51:43.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:43.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj812 testdata.545011 2024-02-05T06:51:43.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:43.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj813 testdata.545011 2024-02-05T06:51:44.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:44.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj814 testdata.545011 2024-02-05T06:51:44.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:44.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj815 testdata.545011 2024-02-05T06:51:44.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:44.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj816 testdata.545011 2024-02-05T06:51:44.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:44.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj817 testdata.545011 2024-02-05T06:51:44.684 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:44.685 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj818 testdata.545011 2024-02-05T06:51:44.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:44.877 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj819 testdata.545011 2024-02-05T06:51:45.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:45.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj820 testdata.545011 2024-02-05T06:51:45.195 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:45.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj821 testdata.545011 2024-02-05T06:51:45.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:45.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj822 testdata.545011 2024-02-05T06:51:45.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:45.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj823 testdata.545011 2024-02-05T06:51:45.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:45.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj824 testdata.545011 2024-02-05T06:51:45.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:45.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj825 testdata.545011 2024-02-05T06:51:45.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:45.941 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj826 testdata.545011 2024-02-05T06:51:46.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:46.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj827 testdata.545011 2024-02-05T06:51:46.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:46.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj828 testdata.545011 2024-02-05T06:51:46.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:46.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj829 testdata.545011 2024-02-05T06:51:46.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:46.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj830 testdata.545011 2024-02-05T06:51:46.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:46.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj831 testdata.545011 2024-02-05T06:51:46.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:46.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj832 testdata.545011 2024-02-05T06:51:47.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:47.055 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj833 testdata.545011 2024-02-05T06:51:47.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:47.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj834 testdata.545011 2024-02-05T06:51:47.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:47.349 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj835 testdata.545011 2024-02-05T06:51:47.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:47.499 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj836 testdata.545011 2024-02-05T06:51:47.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:47.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj837 testdata.545011 2024-02-05T06:51:47.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:47.801 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj838 testdata.545011 2024-02-05T06:51:48.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:48.026 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj839 testdata.545011 2024-02-05T06:51:48.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:48.232 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj840 testdata.545011 2024-02-05T06:51:48.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:48.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj841 testdata.545011 2024-02-05T06:51:48.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:48.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj842 testdata.545011 2024-02-05T06:51:48.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:48.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj843 testdata.545011 2024-02-05T06:51:48.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:48.882 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj844 testdata.545011 2024-02-05T06:51:49.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:49.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj845 testdata.545011 2024-02-05T06:51:49.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:49.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj846 testdata.545011 2024-02-05T06:51:49.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:49.350 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj847 testdata.545011 2024-02-05T06:51:49.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:49.509 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj848 testdata.545011 2024-02-05T06:51:49.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:49.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj849 testdata.545011 2024-02-05T06:51:49.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:49.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj850 testdata.545011 2024-02-05T06:51:49.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:49.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj851 testdata.545011 2024-02-05T06:51:50.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:50.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj852 testdata.545011 2024-02-05T06:51:50.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:50.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj853 testdata.545011 2024-02-05T06:51:50.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:50.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj854 testdata.545011 2024-02-05T06:51:50.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:50.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj855 testdata.545011 2024-02-05T06:51:50.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:50.749 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj856 testdata.545011 2024-02-05T06:51:50.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:50.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj857 testdata.545011 2024-02-05T06:51:51.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:51.060 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj858 testdata.545011 2024-02-05T06:51:51.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:51.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj859 testdata.545011 2024-02-05T06:51:51.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:51.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj860 testdata.545011 2024-02-05T06:51:51.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:51.553 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj861 testdata.545011 2024-02-05T06:51:51.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:51.696 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj862 testdata.545011 2024-02-05T06:51:51.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:51.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj863 testdata.545011 2024-02-05T06:51:52.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:52.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj864 testdata.545011 2024-02-05T06:51:52.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:52.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj865 testdata.545011 2024-02-05T06:51:52.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:52.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj866 testdata.545011 2024-02-05T06:51:52.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:52.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj867 testdata.545011 2024-02-05T06:51:52.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:52.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj868 testdata.545011 2024-02-05T06:51:52.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:52.852 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj869 testdata.545011 2024-02-05T06:51:53.011 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:53.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj870 testdata.545011 2024-02-05T06:51:53.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:53.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj871 testdata.545011 2024-02-05T06:51:53.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:53.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj872 testdata.545011 2024-02-05T06:51:53.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:53.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj873 testdata.545011 2024-02-05T06:51:53.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:53.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj874 testdata.545011 2024-02-05T06:51:53.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:53.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj875 testdata.545011 2024-02-05T06:51:54.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:54.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj876 testdata.545011 2024-02-05T06:51:54.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:54.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj877 testdata.545011 2024-02-05T06:51:54.401 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:54.402 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj878 testdata.545011 2024-02-05T06:51:54.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:54.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj879 testdata.545011 2024-02-05T06:51:54.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:54.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj880 testdata.545011 2024-02-05T06:51:54.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:54.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj881 testdata.545011 2024-02-05T06:51:55.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:55.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj882 testdata.545011 2024-02-05T06:51:55.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:55.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj883 testdata.545011 2024-02-05T06:51:55.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:55.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj884 testdata.545011 2024-02-05T06:51:55.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:55.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj885 testdata.545011 2024-02-05T06:51:55.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:55.683 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj886 testdata.545011 2024-02-05T06:51:55.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:55.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj887 testdata.545011 2024-02-05T06:51:55.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:55.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj888 testdata.545011 2024-02-05T06:51:56.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:56.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj889 testdata.545011 2024-02-05T06:51:56.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:56.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj890 testdata.545011 2024-02-05T06:51:56.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:56.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj891 testdata.545011 2024-02-05T06:51:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:56.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj892 testdata.545011 2024-02-05T06:51:56.755 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:56.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj893 testdata.545011 2024-02-05T06:51:56.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:56.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj894 testdata.545011 2024-02-05T06:51:57.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:57.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj895 testdata.545011 2024-02-05T06:51:57.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:57.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj896 testdata.545011 2024-02-05T06:51:57.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:57.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj897 testdata.545011 2024-02-05T06:51:57.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:57.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj898 testdata.545011 2024-02-05T06:51:57.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:57.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj899 testdata.545011 2024-02-05T06:51:57.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:57.870 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj900 testdata.545011 2024-02-05T06:51:58.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:58.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj901 testdata.545011 2024-02-05T06:51:58.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:58.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj902 testdata.545011 2024-02-05T06:51:58.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:58.373 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj903 testdata.545011 2024-02-05T06:51:58.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:58.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj904 testdata.545011 2024-02-05T06:51:58.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:58.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj905 testdata.545011 2024-02-05T06:51:58.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:58.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj906 testdata.545011 2024-02-05T06:51:59.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:59.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj907 testdata.545011 2024-02-05T06:51:59.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:59.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj908 testdata.545011 2024-02-05T06:51:59.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:59.387 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj909 testdata.545011 2024-02-05T06:51:59.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:59.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj910 testdata.545011 2024-02-05T06:51:59.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:59.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj911 testdata.545011 2024-02-05T06:51:59.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:59.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj912 testdata.545011 2024-02-05T06:51:59.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:51:59.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj913 testdata.545011 2024-02-05T06:52:00.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:00.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj914 testdata.545011 2024-02-05T06:52:00.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:00.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj915 testdata.545011 2024-02-05T06:52:00.416 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:00.417 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj916 testdata.545011 2024-02-05T06:52:00.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:00.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj917 testdata.545011 2024-02-05T06:52:00.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:00.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj918 testdata.545011 2024-02-05T06:52:00.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:00.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj919 testdata.545011 2024-02-05T06:52:01.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:01.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj920 testdata.545011 2024-02-05T06:52:01.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:01.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj921 testdata.545011 2024-02-05T06:52:01.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:01.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj922 testdata.545011 2024-02-05T06:52:01.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:01.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj923 testdata.545011 2024-02-05T06:52:01.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:01.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj924 testdata.545011 2024-02-05T06:52:01.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:01.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj925 testdata.545011 2024-02-05T06:52:01.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:01.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj926 testdata.545011 2024-02-05T06:52:02.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:02.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj927 testdata.545011 2024-02-05T06:52:02.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:02.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj928 testdata.545011 2024-02-05T06:52:02.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:02.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj929 testdata.545011 2024-02-05T06:52:02.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:02.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj930 testdata.545011 2024-02-05T06:52:02.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:02.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj931 testdata.545011 2024-02-05T06:52:02.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:02.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj932 testdata.545011 2024-02-05T06:52:03.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:03.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj933 testdata.545011 2024-02-05T06:52:03.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:03.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj934 testdata.545011 2024-02-05T06:52:03.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:03.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj935 testdata.545011 2024-02-05T06:52:03.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:03.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj936 testdata.545011 2024-02-05T06:52:03.734 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:03.735 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj937 testdata.545011 2024-02-05T06:52:03.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:03.903 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj938 testdata.545011 2024-02-05T06:52:04.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:04.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj939 testdata.545011 2024-02-05T06:52:04.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:04.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj940 testdata.545011 2024-02-05T06:52:04.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:04.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj941 testdata.545011 2024-02-05T06:52:04.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:04.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj942 testdata.545011 2024-02-05T06:52:04.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:04.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj943 testdata.545011 2024-02-05T06:52:04.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:04.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj944 testdata.545011 2024-02-05T06:52:05.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:05.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj945 testdata.545011 2024-02-05T06:52:05.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:05.150 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj946 testdata.545011 2024-02-05T06:52:05.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:05.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj947 testdata.545011 2024-02-05T06:52:05.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:05.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj948 testdata.545011 2024-02-05T06:52:05.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:05.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj949 testdata.545011 2024-02-05T06:52:05.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:05.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj950 testdata.545011 2024-02-05T06:52:05.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:05.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj951 testdata.545011 2024-02-05T06:52:06.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:06.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj952 testdata.545011 2024-02-05T06:52:06.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:06.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj953 testdata.545011 2024-02-05T06:52:06.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:06.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj954 testdata.545011 2024-02-05T06:52:06.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:06.550 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj955 testdata.545011 2024-02-05T06:52:06.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:06.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj956 testdata.545011 2024-02-05T06:52:06.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:06.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj957 testdata.545011 2024-02-05T06:52:07.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:07.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj958 testdata.545011 2024-02-05T06:52:07.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:07.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj959 testdata.545011 2024-02-05T06:52:07.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:07.372 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj960 testdata.545011 2024-02-05T06:52:07.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:07.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj961 testdata.545011 2024-02-05T06:52:07.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:07.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj962 testdata.545011 2024-02-05T06:52:07.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:07.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj963 testdata.545011 2024-02-05T06:52:08.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:08.017 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj964 testdata.545011 2024-02-05T06:52:08.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:08.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj965 testdata.545011 2024-02-05T06:52:08.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:08.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj966 testdata.545011 2024-02-05T06:52:08.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:08.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj967 testdata.545011 2024-02-05T06:52:08.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:08.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj968 testdata.545011 2024-02-05T06:52:08.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:08.829 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj969 testdata.545011 2024-02-05T06:52:08.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:08.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj970 testdata.545011 2024-02-05T06:52:09.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:09.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj971 testdata.545011 2024-02-05T06:52:09.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:09.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj972 testdata.545011 2024-02-05T06:52:09.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:09.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj973 testdata.545011 2024-02-05T06:52:09.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:09.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj974 testdata.545011 2024-02-05T06:52:09.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:09.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj975 testdata.545011 2024-02-05T06:52:09.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:09.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj976 testdata.545011 2024-02-05T06:52:10.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:10.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj977 testdata.545011 2024-02-05T06:52:10.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:10.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj978 testdata.545011 2024-02-05T06:52:10.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:10.388 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj979 testdata.545011 2024-02-05T06:52:10.539 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:10.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj980 testdata.545011 2024-02-05T06:52:10.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:10.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj981 testdata.545011 2024-02-05T06:52:10.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:10.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj982 testdata.545011 2024-02-05T06:52:11.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:11.008 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj983 testdata.545011 2024-02-05T06:52:11.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:11.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj984 testdata.545011 2024-02-05T06:52:11.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:11.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj985 testdata.545011 2024-02-05T06:52:11.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:11.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj986 testdata.545011 2024-02-05T06:52:11.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:11.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj987 testdata.545011 2024-02-05T06:52:11.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:11.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj988 testdata.545011 2024-02-05T06:52:11.980 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:11.981 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj989 testdata.545011 2024-02-05T06:52:12.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:12.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj990 testdata.545011 2024-02-05T06:52:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:12.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj991 testdata.545011 2024-02-05T06:52:12.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:12.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj992 testdata.545011 2024-02-05T06:52:12.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:12.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj993 testdata.545011 2024-02-05T06:52:12.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:12.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj994 testdata.545011 2024-02-05T06:52:12.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:12.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj995 testdata.545011 2024-02-05T06:52:13.086 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:13.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj996 testdata.545011 2024-02-05T06:52:13.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:13.253 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj997 testdata.545011 2024-02-05T06:52:13.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:13.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj998 testdata.545011 2024-02-05T06:52:13.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:13.597 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj999 testdata.545011 2024-02-05T06:52:13.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:52:13.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj1000 testdata.545011 2024-02-05T06:52:13.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rm -f testdata.545011 2024-02-05T06:52:13.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:341: _scrub_abort: get_primary test obj1 2024-02-05T06:52:13.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-02-05T06:52:13.951 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:52:13.952 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-02-05T06:52:13.952 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:52:14.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:341: _scrub_abort: local primary=1 2024-02-05T06:52:14.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:342: _scrub_abort: local pgid=1.0 2024-02-05T06:52:14.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: ceph tell 1.0 schedule-deep-scrub 2024-02-05T06:52:14.419 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:52:14.420 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": true, 2024-02-05T06:52:14.420 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:52:14.420 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-22T06:50:34.413289+0000" 2024-02-05T06:52:14.420 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:52:14.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: set -o pipefail 2024-02-05T06:52:14.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:348: _scrub_abort: found=no 2024-02-05T06:52:14.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: seq 0 200 2024-02-05T06:52:14.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:52:14.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: flush_pg_stats 2024-02-05T06:52:14.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:52:14.436 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:52:14.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:52:14.742 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:52:14.742 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:52:14.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:52:14.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:14.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836522 2024-02-05T06:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836522 2024-02-05T06:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522' 2024-02-05T06:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:14.906 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:52:15.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672999 2024-02-05T06:52:15.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672999 2024-02-05T06:52:15.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-42949672999' 2024-02-05T06:52:15.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:15.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:52:15.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509477 2024-02-05T06:52:15.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509477 2024-02-05T06:52:15.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-42949672999 2-64424509477' 2024-02-05T06:52:15.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:15.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836522 2024-02-05T06:52:15.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:15.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:52:15.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836522 2024-02-05T06:52:15.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:15.181 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836522 2024-02-05T06:52:15.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836522 2024-02-05T06:52:15.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-02-05T06:52:15.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:15.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836522 2024-02-05T06:52:15.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:16.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:52:16.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:16.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836522 -lt 21474836522 2024-02-05T06:52:16.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:16.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:16.823 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672999 2024-02-05T06:52:16.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:52:16.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672999 2024-02-05T06:52:16.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:16.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672999 2024-02-05T06:52:16.827 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672999 2024-02-05T06:52:16.827 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2024-02-05T06:52:16.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:52:17.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673000 -lt 42949672999 2024-02-05T06:52:17.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:17.146 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509477 2024-02-05T06:52:17.146 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:17.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:52:17.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509477 2024-02-05T06:52:17.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:17.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509477 2024-02-05T06:52:17.149 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509477 2024-02-05T06:52:17.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509477' 2024-02-05T06:52:17.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:52:17.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509477 -lt 64424509477 2024-02-05T06:52:17.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: ceph pg dump pgs 2024-02-05T06:52:17.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: grep -q scrubbing 2024-02-05T06:52:17.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: grep '^1.0' 2024-02-05T06:52:17.761 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:52:17.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: found=yes 2024-02-05T06:52:17.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:356: _scrub_abort: break 2024-02-05T06:52:17.778 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:359: _scrub_abort: set +o pipefail 2024-02-05T06:52:17.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: test yes = no 2024-02-05T06:52:17.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:367: _scrub_abort: ceph osd set nodeep-scrub 2024-02-05T06:52:18.262 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T06:52:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:368: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2024-02-05T06:52:18.280 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: ceph osd set noscrub 2024-02-05T06:52:18.823 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:52:18.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:374: _scrub_abort: set -o pipefail 2024-02-05T06:52:18.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: seq 0 200 2024-02-05T06:52:18.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:52:18.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:52:18.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:52:18.846 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:52:19.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:52:19.158 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:52:19.158 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:52:19.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:52:19.158 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:19.159 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:52:19.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836524 2024-02-05T06:52:19.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836524 2024-02-05T06:52:19.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524' 2024-02-05T06:52:19.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:19.318 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:52:19.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673001 2024-02-05T06:52:19.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673001 2024-02-05T06:52:19.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-42949673001' 2024-02-05T06:52:19.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:19.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:52:19.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509479 2024-02-05T06:52:19.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509479 2024-02-05T06:52:19.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-42949673001 2-64424509479' 2024-02-05T06:52:19.600 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:19.600 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836524 2024-02-05T06:52:19.600 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:19.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:52:19.602 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836524 2024-02-05T06:52:19.602 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:19.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836524 2024-02-05T06:52:19.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-02-05T06:52:19.603 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836524 2024-02-05T06:52:19.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:19.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836522 -lt 21474836524 2024-02-05T06:52:19.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:20.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:52:20.908 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:21.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836524 -lt 21474836524 2024-02-05T06:52:21.221 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:21.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673001 2024-02-05T06:52:21.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:21.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:52:21.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673001 2024-02-05T06:52:21.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:21.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673001 2024-02-05T06:52:21.226 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673001 2024-02-05T06:52:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-02-05T06:52:21.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:52:21.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673001 -lt 42949673001 2024-02-05T06:52:21.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:21.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509479 2024-02-05T06:52:21.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:21.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:52:21.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509479 2024-02-05T06:52:21.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:21.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509479 2024-02-05T06:52:21.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2024-02-05T06:52:21.529 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509479 2024-02-05T06:52:21.530 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:52:21.878 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509479 -lt 64424509479 2024-02-05T06:52:21.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:52:21.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:52:21.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:52:22.183 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:52:22.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:52:22.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:52:22.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:52:22.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:52:22.201 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:52:22.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:52:22.507 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:52:22.507 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:52:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:52:22.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:22.508 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:52:22.646 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836525 2024-02-05T06:52:22.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836525 2024-02-05T06:52:22.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525' 2024-02-05T06:52:22.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:22.647 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:52:22.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673003 2024-02-05T06:52:22.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673003 2024-02-05T06:52:22.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-42949673003' 2024-02-05T06:52:22.774 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:22.774 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:52:22.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509480 2024-02-05T06:52:22.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509480 2024-02-05T06:52:22.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-42949673003 2-64424509480' 2024-02-05T06:52:22.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:22.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836525 2024-02-05T06:52:22.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:22.912 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:52:22.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836525 2024-02-05T06:52:22.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:22.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836525 2024-02-05T06:52:22.914 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836525 2024-02-05T06:52:22.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-02-05T06:52:22.915 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:23.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836524 -lt 21474836525 2024-02-05T06:52:23.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:24.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:52:24.229 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:24.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836525 2024-02-05T06:52:24.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:24.570 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673003 2024-02-05T06:52:24.570 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:24.571 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:52:24.572 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673003 2024-02-05T06:52:24.572 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:24.573 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673003 2024-02-05T06:52:24.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673003 2024-02-05T06:52:24.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2024-02-05T06:52:24.574 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:52:24.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673003 -lt 42949673003 2024-02-05T06:52:24.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:24.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:24.900 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509480 2024-02-05T06:52:24.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:52:24.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509480 2024-02-05T06:52:24.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:24.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509480 2024-02-05T06:52:24.904 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509480 2024-02-05T06:52:24.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509480' 2024-02-05T06:52:24.905 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:52:25.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509481 -lt 64424509480 2024-02-05T06:52:25.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:52:25.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:52:25.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:52:25.501 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:52:25.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:52:25.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:52:25.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:52:25.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:52:25.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:52:25.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:52:25.809 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:52:25.810 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:52:25.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:52:25.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:25.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:52:25.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836527 2024-02-05T06:52:25.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836527 2024-02-05T06:52:25.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527' 2024-02-05T06:52:25.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:25.946 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:52:26.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673005 2024-02-05T06:52:26.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673005 2024-02-05T06:52:26.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-42949673005' 2024-02-05T06:52:26.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:26.073 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:52:26.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509482 2024-02-05T06:52:26.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509482 2024-02-05T06:52:26.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-42949673005 2-64424509482' 2024-02-05T06:52:26.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:26.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836527 2024-02-05T06:52:26.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:26.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:52:26.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836527 2024-02-05T06:52:26.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:26.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836527 2024-02-05T06:52:26.204 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836527 2024-02-05T06:52:26.205 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-02-05T06:52:26.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:26.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836526 -lt 21474836527 2024-02-05T06:52:26.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:27.511 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:52:27.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:27.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836526 -lt 21474836527 2024-02-05T06:52:27.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:28.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:52:28.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:29.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836527 2024-02-05T06:52:29.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:29.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673005 2024-02-05T06:52:29.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:29.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:52:29.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673005 2024-02-05T06:52:29.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:29.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673005 2024-02-05T06:52:29.127 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673005 2024-02-05T06:52:29.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673005' 2024-02-05T06:52:29.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:52:29.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673005 -lt 42949673005 2024-02-05T06:52:29.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:29.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509482 2024-02-05T06:52:29.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:29.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:52:29.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509482 2024-02-05T06:52:29.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:29.469 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509482 2024-02-05T06:52:29.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509482 2024-02-05T06:52:29.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2024-02-05T06:52:29.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:52:29.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509482 -lt 64424509482 2024-02-05T06:52:29.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:52:29.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:52:29.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:52:30.124 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:52:30.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:52:30.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:52:30.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:52:30.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:52:30.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:52:30.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:52:30.453 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:52:30.453 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:52:30.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:52:30.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:30.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:52:30.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836529 2024-02-05T06:52:30.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836529 2024-02-05T06:52:30.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529' 2024-02-05T06:52:30.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:30.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:52:30.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673007 2024-02-05T06:52:30.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673007 2024-02-05T06:52:30.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-42949673007' 2024-02-05T06:52:30.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:30.727 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:52:30.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509484 2024-02-05T06:52:30.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509484 2024-02-05T06:52:30.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-42949673007 2-64424509484' 2024-02-05T06:52:30.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:30.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836529 2024-02-05T06:52:30.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:30.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:52:30.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836529 2024-02-05T06:52:30.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:30.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836529 2024-02-05T06:52:30.861 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836529 2024-02-05T06:52:30.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-02-05T06:52:30.861 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:31.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836529 2024-02-05T06:52:31.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:32.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:52:32.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:32.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836529 2024-02-05T06:52:32.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:32.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:32.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673007 2024-02-05T06:52:32.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:52:32.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673007 2024-02-05T06:52:32.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:32.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673007 2024-02-05T06:52:32.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673007' 2024-02-05T06:52:32.459 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673007 2024-02-05T06:52:32.459 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:52:32.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673007 -lt 42949673007 2024-02-05T06:52:32.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:32.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:32.743 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509484 2024-02-05T06:52:32.744 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:52:32.745 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509484 2024-02-05T06:52:32.745 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:32.746 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509484 2024-02-05T06:52:32.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509484 2024-02-05T06:52:32.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509484' 2024-02-05T06:52:32.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:52:33.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509484 -lt 64424509484 2024-02-05T06:52:33.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:52:33.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:52:33.052 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:52:33.319 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:52:33.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:52:33.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:52:33.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:52:33.335 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:52:33.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:52:33.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:52:33.625 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:52:33.625 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:52:33.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:52:33.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:33.626 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:52:33.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836530 2024-02-05T06:52:33.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836530 2024-02-05T06:52:33.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530' 2024-02-05T06:52:33.766 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:33.766 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:52:33.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673008 2024-02-05T06:52:33.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673008 2024-02-05T06:52:33.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-42949673008' 2024-02-05T06:52:33.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:33.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:52:34.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509486 2024-02-05T06:52:34.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509486 2024-02-05T06:52:34.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-42949673008 2-64424509486' 2024-02-05T06:52:34.037 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:34.038 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836530 2024-02-05T06:52:34.038 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:34.038 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:52:34.039 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836530 2024-02-05T06:52:34.039 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:34.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836530 2024-02-05T06:52:34.040 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836530 2024-02-05T06:52:34.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-02-05T06:52:34.040 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:34.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836530 2024-02-05T06:52:34.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:34.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673008 2024-02-05T06:52:34.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:34.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:52:34.332 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673008 2024-02-05T06:52:34.333 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:34.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673008 2024-02-05T06:52:34.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2024-02-05T06:52:34.334 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673008 2024-02-05T06:52:34.334 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:52:34.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673007 -lt 42949673008 2024-02-05T06:52:34.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:35.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:52:35.660 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:52:35.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673009 -lt 42949673008 2024-02-05T06:52:35.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:35.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509486 2024-02-05T06:52:35.973 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:35.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:52:35.974 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509486 2024-02-05T06:52:35.975 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:35.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509486 2024-02-05T06:52:35.976 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509486' 2024-02-05T06:52:35.976 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509486 2024-02-05T06:52:35.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:52:36.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509486 -lt 64424509486 2024-02-05T06:52:36.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:52:36.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:52:36.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:52:36.575 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:52:36.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:52:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:52:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:52:36.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:52:36.593 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:52:36.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:52:36.896 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:52:36.896 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:52:36.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:52:36.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:36.896 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:52:37.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836532 2024-02-05T06:52:37.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836532 2024-02-05T06:52:37.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532' 2024-02-05T06:52:37.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:37.061 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:52:37.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673010 2024-02-05T06:52:37.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673010 2024-02-05T06:52:37.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-42949673010' 2024-02-05T06:52:37.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:37.187 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:52:37.316 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509487 2024-02-05T06:52:37.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509487 2024-02-05T06:52:37.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-42949673010 2-64424509487' 2024-02-05T06:52:37.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:37.317 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836532 2024-02-05T06:52:37.317 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:37.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:52:37.318 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836532 2024-02-05T06:52:37.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:37.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836532 2024-02-05T06:52:37.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-02-05T06:52:37.320 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836532 2024-02-05T06:52:37.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:37.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836532 2024-02-05T06:52:37.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:38.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:52:38.608 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:38.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836532 2024-02-05T06:52:38.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:38.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673010 2024-02-05T06:52:38.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:38.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:52:38.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673010 2024-02-05T06:52:38.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:38.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673010 2024-02-05T06:52:38.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2024-02-05T06:52:38.913 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673010 2024-02-05T06:52:38.914 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:52:39.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673010 -lt 42949673010 2024-02-05T06:52:39.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:39.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509487 2024-02-05T06:52:39.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:39.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:52:39.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509487 2024-02-05T06:52:39.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:39.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509487 2024-02-05T06:52:39.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509487' 2024-02-05T06:52:39.247 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509487 2024-02-05T06:52:39.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:52:39.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509487 -lt 64424509487 2024-02-05T06:52:39.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:52:39.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:52:39.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:52:39.914 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:52:39.930 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:52:39.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:52:39.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:52:39.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:52:39.931 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:52:40.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:52:40.234 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:52:40.234 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:52:40.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:52:40.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:40.235 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:52:40.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836534 2024-02-05T06:52:40.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836534 2024-02-05T06:52:40.399 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534' 2024-02-05T06:52:40.400 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:40.400 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:52:40.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673011 2024-02-05T06:52:40.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673011 2024-02-05T06:52:40.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-42949673011' 2024-02-05T06:52:40.549 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:40.549 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:52:40.676 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509489 2024-02-05T06:52:40.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509489 2024-02-05T06:52:40.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-42949673011 2-64424509489' 2024-02-05T06:52:40.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:40.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836534 2024-02-05T06:52:40.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:40.678 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:52:40.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836534 2024-02-05T06:52:40.679 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:40.679 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836534 2024-02-05T06:52:40.680 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836534 2024-02-05T06:52:40.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-02-05T06:52:40.680 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:40.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836533 -lt 21474836534 2024-02-05T06:52:40.964 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:41.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:52:41.967 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:42.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836534 2024-02-05T06:52:42.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:42.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673011 2024-02-05T06:52:42.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:42.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:52:42.476 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673011 2024-02-05T06:52:42.476 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:42.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673011 2024-02-05T06:52:42.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2024-02-05T06:52:42.479 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673011 2024-02-05T06:52:42.480 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:52:42.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673012 -lt 42949673011 2024-02-05T06:52:42.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:42.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509489 2024-02-05T06:52:42.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:42.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:52:42.809 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:42.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509489 2024-02-05T06:52:42.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509489 2024-02-05T06:52:42.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509489' 2024-02-05T06:52:42.812 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509489 2024-02-05T06:52:42.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:52:43.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509489 -lt 64424509489 2024-02-05T06:52:43.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:52:43.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:52:43.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:52:43.409 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:52:43.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:52:43.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:52:43.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:52:43.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:52:43.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:52:43.772 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:52:43.773 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:52:43.773 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:52:43.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:52:43.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:43.773 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:52:43.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836535 2024-02-05T06:52:43.971 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836535 2024-02-05T06:52:43.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535' 2024-02-05T06:52:43.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:43.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:52:44.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673013 2024-02-05T06:52:44.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673013 2024-02-05T06:52:44.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-42949673013' 2024-02-05T06:52:44.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:44.116 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:52:44.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509491 2024-02-05T06:52:44.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509491 2024-02-05T06:52:44.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-42949673013 2-64424509491' 2024-02-05T06:52:44.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:44.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836535 2024-02-05T06:52:44.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:44.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:52:44.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836535 2024-02-05T06:52:44.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:44.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836535 2024-02-05T06:52:44.247 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836535 2024-02-05T06:52:44.248 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-02-05T06:52:44.248 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:44.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836535 2024-02-05T06:52:44.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:45.589 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:52:45.590 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:45.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836535 2024-02-05T06:52:45.892 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:46.894 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:52:46.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:47.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836535 2024-02-05T06:52:47.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:47.201 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673013 2024-02-05T06:52:47.201 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:47.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:52:47.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673013 2024-02-05T06:52:47.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:47.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673013 2024-02-05T06:52:47.204 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673013 2024-02-05T06:52:47.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673013' 2024-02-05T06:52:47.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:52:47.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673014 -lt 42949673013 2024-02-05T06:52:47.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:47.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509491 2024-02-05T06:52:47.541 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:47.543 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:52:47.544 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509491 2024-02-05T06:52:47.544 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:47.545 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509491 2024-02-05T06:52:47.545 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509491 2024-02-05T06:52:47.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509491' 2024-02-05T06:52:47.546 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:52:47.853 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509491 -lt 64424509491 2024-02-05T06:52:47.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:52:47.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:52:47.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:52:48.166 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:52:48.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:52:48.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:52:48.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:52:48.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:52:48.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:52:48.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:52:48.473 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:52:48.473 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:52:48.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:52:48.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:48.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:52:48.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836537 2024-02-05T06:52:48.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836537 2024-02-05T06:52:48.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537' 2024-02-05T06:52:48.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:48.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:52:48.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673015 2024-02-05T06:52:48.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673015 2024-02-05T06:52:48.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-42949673015' 2024-02-05T06:52:48.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:48.753 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:52:48.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509493 2024-02-05T06:52:48.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509493 2024-02-05T06:52:48.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-42949673015 2-64424509493' 2024-02-05T06:52:48.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:48.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836537 2024-02-05T06:52:48.885 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:48.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:52:48.887 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836537 2024-02-05T06:52:48.887 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:48.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836537 2024-02-05T06:52:48.888 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836537 2024-02-05T06:52:48.888 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-02-05T06:52:48.888 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:49.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836537 2024-02-05T06:52:49.190 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:50.192 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:52:50.193 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:50.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836537 2024-02-05T06:52:50.517 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:50.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673015 2024-02-05T06:52:50.518 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:50.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:52:50.520 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673015 2024-02-05T06:52:50.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:50.521 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673015 2024-02-05T06:52:50.522 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673015 2024-02-05T06:52:50.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673015' 2024-02-05T06:52:50.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:52:50.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673015 -lt 42949673015 2024-02-05T06:52:50.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:50.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:50.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509493 2024-02-05T06:52:50.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:52:50.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509493 2024-02-05T06:52:50.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:50.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509493 2024-02-05T06:52:50.816 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509493 2024-02-05T06:52:50.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509493' 2024-02-05T06:52:50.817 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:52:51.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509493 -lt 64424509493 2024-02-05T06:52:51.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:52:51.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:52:51.118 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:52:51.415 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:52:51.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:52:51.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:52:51.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:52:51.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:52:51.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:52:51.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:52:51.719 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:52:51.719 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:52:51.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:52:51.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:51.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:52:51.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836539 2024-02-05T06:52:51.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836539 2024-02-05T06:52:51.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539' 2024-02-05T06:52:51.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:51.913 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:52:52.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673017 2024-02-05T06:52:52.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673017 2024-02-05T06:52:52.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-42949673017' 2024-02-05T06:52:52.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:52.048 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:52:52.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509494 2024-02-05T06:52:52.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509494 2024-02-05T06:52:52.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-64424509494' 2024-02-05T06:52:52.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:52.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836539 2024-02-05T06:52:52.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:52.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:52:52.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836539 2024-02-05T06:52:52.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:52.182 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836539 2024-02-05T06:52:52.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836539 2024-02-05T06:52:52.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2024-02-05T06:52:52.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:52.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836539 2024-02-05T06:52:52.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:53.476 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:52:53.477 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:53.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836539 2024-02-05T06:52:53.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:54.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:52:54.784 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:55.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836539 2024-02-05T06:52:55.095 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:55.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673017 2024-02-05T06:52:55.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:55.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:52:55.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673017 2024-02-05T06:52:55.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:55.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673017 2024-02-05T06:52:55.099 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673017 2024-02-05T06:52:55.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673017' 2024-02-05T06:52:55.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:52:55.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673017 -lt 42949673017 2024-02-05T06:52:55.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:55.421 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:55.421 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509494 2024-02-05T06:52:55.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:52:55.423 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509494 2024-02-05T06:52:55.423 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:55.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509494 2024-02-05T06:52:55.425 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509494 2024-02-05T06:52:55.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509494' 2024-02-05T06:52:55.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:52:55.760 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509495 -lt 64424509494 2024-02-05T06:52:55.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:52:55.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:52:55.761 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:52:56.103 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:52:56.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:52:56.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:52:56.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:52:56.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:52:56.121 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:52:56.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:52:56.425 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:52:56.425 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:52:56.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:52:56.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:56.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:52:56.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836541 2024-02-05T06:52:56.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836541 2024-02-05T06:52:56.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541' 2024-02-05T06:52:56.569 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:56.569 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:52:56.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673019 2024-02-05T06:52:56.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673019 2024-02-05T06:52:56.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-42949673019' 2024-02-05T06:52:56.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:56.708 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:52:56.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509496 2024-02-05T06:52:56.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509496 2024-02-05T06:52:56.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-42949673019 2-64424509496' 2024-02-05T06:52:56.847 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:56.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836541 2024-02-05T06:52:56.847 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:56.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:52:56.849 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836541 2024-02-05T06:52:56.849 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:56.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836541 2024-02-05T06:52:56.850 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836541 2024-02-05T06:52:56.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-02-05T06:52:56.850 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:57.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836541 2024-02-05T06:52:57.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:52:58.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:52:58.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:52:58.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836541 2024-02-05T06:52:58.462 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:58.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673019 2024-02-05T06:52:58.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:58.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:52:58.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673019 2024-02-05T06:52:58.465 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:58.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673019 2024-02-05T06:52:58.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673019' 2024-02-05T06:52:58.466 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673019 2024-02-05T06:52:58.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:52:58.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673019 -lt 42949673019 2024-02-05T06:52:58.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:52:58.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509496 2024-02-05T06:52:58.754 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:52:58.756 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:52:58.757 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509496 2024-02-05T06:52:58.757 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:52:58.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509496 2024-02-05T06:52:58.758 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509496 2024-02-05T06:52:58.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2024-02-05T06:52:58.759 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:52:59.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509496 -lt 64424509496 2024-02-05T06:52:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:52:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:52:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:52:59.379 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:52:59.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:52:59.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:52:59.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:52:59.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:52:59.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:52:59.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:52:59.698 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:52:59.698 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:52:59.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:52:59.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:59.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:52:59.858 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836543 2024-02-05T06:52:59.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836543 2024-02-05T06:52:59.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543' 2024-02-05T06:52:59.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:59.859 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:52:59.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673020 2024-02-05T06:52:59.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673020 2024-02-05T06:52:59.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-42949673020' 2024-02-05T06:52:59.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:52:59.994 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:00.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509498 2024-02-05T06:53:00.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509498 2024-02-05T06:53:00.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-42949673020 2-64424509498' 2024-02-05T06:53:00.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:00.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836543 2024-02-05T06:53:00.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:00.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:00.129 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:00.129 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836543 2024-02-05T06:53:00.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836543 2024-02-05T06:53:00.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-02-05T06:53:00.130 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836543 2024-02-05T06:53:00.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:00.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836543 2024-02-05T06:53:00.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:01.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:01.420 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:01.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836543 2024-02-05T06:53:01.724 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:02.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:53:02.727 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:03.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836543 2024-02-05T06:53:03.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:03.020 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673020 2024-02-05T06:53:03.020 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:03.022 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:03.022 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673020 2024-02-05T06:53:03.023 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:03.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673020 2024-02-05T06:53:03.024 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673020 2024-02-05T06:53:03.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-02-05T06:53:03.024 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:03.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673021 -lt 42949673020 2024-02-05T06:53:03.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:03.363 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509498 2024-02-05T06:53:03.363 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:03.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:03.365 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509498 2024-02-05T06:53:03.365 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:03.367 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509498 2024-02-05T06:53:03.367 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509498 2024-02-05T06:53:03.368 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2024-02-05T06:53:03.368 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:03.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509498 -lt 64424509498 2024-02-05T06:53:03.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:03.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:03.666 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:03.968 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:03.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:03.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:03.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:03.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:03.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:04.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:04.292 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:04.292 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:04.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:04.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:04.292 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:04.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836545 2024-02-05T06:53:04.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836545 2024-02-05T06:53:04.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545' 2024-02-05T06:53:04.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:04.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:04.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673022 2024-02-05T06:53:04.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673022 2024-02-05T06:53:04.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-42949673022' 2024-02-05T06:53:04.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:04.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:04.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509500 2024-02-05T06:53:04.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509500 2024-02-05T06:53:04.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-42949673022 2-64424509500' 2024-02-05T06:53:04.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:04.739 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836545 2024-02-05T06:53:04.739 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:04.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:04.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836545 2024-02-05T06:53:04.740 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:04.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836545 2024-02-05T06:53:04.741 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836545 2024-02-05T06:53:04.742 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2024-02-05T06:53:04.742 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:05.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836545 2024-02-05T06:53:05.027 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:06.029 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:06.029 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:06.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836545 2024-02-05T06:53:06.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:06.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673022 2024-02-05T06:53:06.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:06.326 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:06.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673022 2024-02-05T06:53:06.328 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:06.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673022 2024-02-05T06:53:06.329 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673022 2024-02-05T06:53:06.330 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673022' 2024-02-05T06:53:06.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:06.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673023 -lt 42949673022 2024-02-05T06:53:06.663 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:06.663 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509500 2024-02-05T06:53:06.664 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:06.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:06.666 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:06.666 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509500 2024-02-05T06:53:06.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509500 2024-02-05T06:53:06.667 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509500' 2024-02-05T06:53:06.667 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509500 2024-02-05T06:53:06.668 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:06.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509500 -lt 64424509500 2024-02-05T06:53:06.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:06.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:06.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:07.255 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:07.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:07.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:07.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:07.272 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:07.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:07.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:07.594 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:07.594 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:07.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:07.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:07.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:07.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836546 2024-02-05T06:53:07.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836546 2024-02-05T06:53:07.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546' 2024-02-05T06:53:07.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:07.780 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:07.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673024 2024-02-05T06:53:07.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673024 2024-02-05T06:53:07.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673024' 2024-02-05T06:53:07.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:07.923 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:08.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509501 2024-02-05T06:53:08.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509501 2024-02-05T06:53:08.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673024 2-64424509501' 2024-02-05T06:53:08.057 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:08.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836546 2024-02-05T06:53:08.058 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:08.058 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:08.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836546 2024-02-05T06:53:08.059 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:08.061 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836546 2024-02-05T06:53:08.061 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836546 2024-02-05T06:53:08.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-02-05T06:53:08.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:08.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836546 -lt 21474836546 2024-02-05T06:53:08.337 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:08.338 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673024 2024-02-05T06:53:08.338 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:08.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:08.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673024 2024-02-05T06:53:08.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:08.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673024 2024-02-05T06:53:08.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673024' 2024-02-05T06:53:08.342 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673024 2024-02-05T06:53:08.342 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:08.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673023 -lt 42949673024 2024-02-05T06:53:08.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:09.655 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:09.656 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:09.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673024 -lt 42949673024 2024-02-05T06:53:09.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:09.955 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:09.955 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509501 2024-02-05T06:53:09.955 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:09.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509501 2024-02-05T06:53:09.956 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:09.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509501 2024-02-05T06:53:09.958 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509501 2024-02-05T06:53:09.959 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509501' 2024-02-05T06:53:09.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:10.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509502 -lt 64424509501 2024-02-05T06:53:10.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:10.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:10.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:10.616 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:10.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:10.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:10.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:10.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:10.634 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:10.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:10.918 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:10.918 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:10.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:10.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:10.918 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:11.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836548 2024-02-05T06:53:11.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836548 2024-02-05T06:53:11.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548' 2024-02-05T06:53:11.091 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:11.091 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:11.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673026 2024-02-05T06:53:11.234 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673026 2024-02-05T06:53:11.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673026' 2024-02-05T06:53:11.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:11.235 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:11.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509503 2024-02-05T06:53:11.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509503 2024-02-05T06:53:11.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673026 2-64424509503' 2024-02-05T06:53:11.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:11.362 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836548 2024-02-05T06:53:11.362 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:11.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:11.363 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836548 2024-02-05T06:53:11.363 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:11.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836548 2024-02-05T06:53:11.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-02-05T06:53:11.364 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836548 2024-02-05T06:53:11.365 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:11.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836547 -lt 21474836548 2024-02-05T06:53:11.642 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:12.644 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:12.644 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:12.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836548 -lt 21474836548 2024-02-05T06:53:12.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:12.955 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673026 2024-02-05T06:53:12.955 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:12.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:12.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673026 2024-02-05T06:53:12.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:12.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673026 2024-02-05T06:53:12.958 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673026' 2024-02-05T06:53:12.958 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673026 2024-02-05T06:53:12.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:13.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673026 -lt 42949673026 2024-02-05T06:53:13.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:13.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509503 2024-02-05T06:53:13.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:13.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:13.297 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509503 2024-02-05T06:53:13.297 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:13.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509503 2024-02-05T06:53:13.298 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-02-05T06:53:13.298 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509503 2024-02-05T06:53:13.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:13.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509503 -lt 64424509503 2024-02-05T06:53:13.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:13.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:13.590 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:13.880 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:13.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:13.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:13.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:13.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:13.896 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:14.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:14.177 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:14.177 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:14.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:14.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:14.178 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:14.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836550 2024-02-05T06:53:14.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836550 2024-02-05T06:53:14.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550' 2024-02-05T06:53:14.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:14.346 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:14.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673027 2024-02-05T06:53:14.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673027 2024-02-05T06:53:14.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673027' 2024-02-05T06:53:14.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:14.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:14.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509505 2024-02-05T06:53:14.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509505 2024-02-05T06:53:14.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673027 2-64424509505' 2024-02-05T06:53:14.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:14.611 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836550 2024-02-05T06:53:14.612 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:14.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:14.613 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836550 2024-02-05T06:53:14.613 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:14.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836550 2024-02-05T06:53:14.614 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836550 2024-02-05T06:53:14.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-02-05T06:53:14.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:14.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836548 -lt 21474836550 2024-02-05T06:53:14.913 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:15.914 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:15.916 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:16.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836550 2024-02-05T06:53:16.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:16.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:16.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673027 2024-02-05T06:53:16.214 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:16.215 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673027 2024-02-05T06:53:16.215 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:16.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673027 2024-02-05T06:53:16.217 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673027 2024-02-05T06:53:16.217 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673027' 2024-02-05T06:53:16.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:16.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673028 -lt 42949673027 2024-02-05T06:53:16.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:16.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509505 2024-02-05T06:53:16.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:16.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:16.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509505 2024-02-05T06:53:16.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:16.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509505 2024-02-05T06:53:16.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-02-05T06:53:16.533 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509505 2024-02-05T06:53:16.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:16.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509505 -lt 64424509505 2024-02-05T06:53:16.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:16.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:16.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:17.123 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:17.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:17.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:17.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:17.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:17.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:17.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:17.446 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:17.446 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:17.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:17.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:17.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:17.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836551 2024-02-05T06:53:17.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836551 2024-02-05T06:53:17.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551' 2024-02-05T06:53:17.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:17.608 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:17.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673029 2024-02-05T06:53:17.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673029 2024-02-05T06:53:17.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673029' 2024-02-05T06:53:17.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:17.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:17.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509506 2024-02-05T06:53:17.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509506 2024-02-05T06:53:17.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673029 2-64424509506' 2024-02-05T06:53:17.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:17.915 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836551 2024-02-05T06:53:17.915 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:17.916 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:17.917 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836551 2024-02-05T06:53:17.917 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:17.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836551 2024-02-05T06:53:17.918 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836551 2024-02-05T06:53:17.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836551' 2024-02-05T06:53:17.918 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:18.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836551 -lt 21474836551 2024-02-05T06:53:18.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:18.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673029 2024-02-05T06:53:18.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:18.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:18.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673029 2024-02-05T06:53:18.206 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:18.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673029 2024-02-05T06:53:18.207 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673029' 2024-02-05T06:53:18.207 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673029 2024-02-05T06:53:18.208 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:18.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673029 -lt 42949673029 2024-02-05T06:53:18.557 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:18.558 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509506 2024-02-05T06:53:18.559 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:18.560 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:18.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509506 2024-02-05T06:53:18.562 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:18.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509506 2024-02-05T06:53:18.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509506' 2024-02-05T06:53:18.563 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509506 2024-02-05T06:53:18.563 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:18.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509505 -lt 64424509506 2024-02-05T06:53:18.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:19.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:19.858 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:20.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509507 -lt 64424509506 2024-02-05T06:53:20.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:20.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:20.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:20.472 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:20.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:20.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:20.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:20.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:20.489 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:20.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:20.792 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:20.792 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:20.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:20.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:20.793 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:20.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836553 2024-02-05T06:53:20.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836553 2024-02-05T06:53:20.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553' 2024-02-05T06:53:20.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:20.957 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:21.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673031 2024-02-05T06:53:21.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673031 2024-02-05T06:53:21.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673031' 2024-02-05T06:53:21.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:21.089 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:21.222 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509508 2024-02-05T06:53:21.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509508 2024-02-05T06:53:21.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673031 2-64424509508' 2024-02-05T06:53:21.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:21.223 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836553 2024-02-05T06:53:21.224 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:21.224 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:21.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836553 2024-02-05T06:53:21.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836553 2024-02-05T06:53:21.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-02-05T06:53:21.227 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836553 2024-02-05T06:53:21.227 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:21.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836552 -lt 21474836553 2024-02-05T06:53:21.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:22.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:22.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:22.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836553 -lt 21474836553 2024-02-05T06:53:22.831 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:22.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673031 2024-02-05T06:53:22.832 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:22.834 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:22.835 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673031 2024-02-05T06:53:22.835 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:22.836 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673031 2024-02-05T06:53:22.836 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673031 2024-02-05T06:53:22.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-02-05T06:53:22.837 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:23.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673031 -lt 42949673031 2024-02-05T06:53:23.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:23.173 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509508 2024-02-05T06:53:23.173 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:23.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:23.175 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509508 2024-02-05T06:53:23.176 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:23.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509508 2024-02-05T06:53:23.177 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509508 2024-02-05T06:53:23.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509508' 2024-02-05T06:53:23.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:23.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509508 -lt 64424509508 2024-02-05T06:53:23.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:23.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:23.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:23.790 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:23.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:23.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:23.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:23.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:23.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:24.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:24.106 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:24.106 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:24.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:24.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:24.107 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:24.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836555 2024-02-05T06:53:24.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836555 2024-02-05T06:53:24.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555' 2024-02-05T06:53:24.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:24.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:24.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673032 2024-02-05T06:53:24.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673032 2024-02-05T06:53:24.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673032' 2024-02-05T06:53:24.398 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:24.398 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:24.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509510 2024-02-05T06:53:24.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509510 2024-02-05T06:53:24.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673032 2-64424509510' 2024-02-05T06:53:24.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:24.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836555 2024-02-05T06:53:24.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:24.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:24.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836555 2024-02-05T06:53:24.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:24.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836555 2024-02-05T06:53:24.534 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836555 2024-02-05T06:53:24.535 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836555' 2024-02-05T06:53:24.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:24.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836553 -lt 21474836555 2024-02-05T06:53:24.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:25.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:25.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:26.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836555 -lt 21474836555 2024-02-05T06:53:26.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:26.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:26.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673032 2024-02-05T06:53:26.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:26.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673032 2024-02-05T06:53:26.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:26.128 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673032 2024-02-05T06:53:26.128 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673032 2024-02-05T06:53:26.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673032' 2024-02-05T06:53:26.129 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:26.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673033 -lt 42949673032 2024-02-05T06:53:26.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:26.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:26.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509510 2024-02-05T06:53:26.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:26.456 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509510 2024-02-05T06:53:26.456 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:26.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509510 2024-02-05T06:53:26.457 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509510 2024-02-05T06:53:26.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-02-05T06:53:26.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:26.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509510 -lt 64424509510 2024-02-05T06:53:26.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:26.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:26.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:27.043 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:27.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:27.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:27.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:27.059 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:27.060 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:27.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:27.362 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:27.362 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:27.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:27.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:27.362 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:27.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836556 2024-02-05T06:53:27.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836556 2024-02-05T06:53:27.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556' 2024-02-05T06:53:27.544 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:27.544 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673034 2024-02-05T06:53:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673034 2024-02-05T06:53:27.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673034' 2024-02-05T06:53:27.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:27.705 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:27.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509511 2024-02-05T06:53:27.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509511 2024-02-05T06:53:27.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673034 2-64424509511' 2024-02-05T06:53:27.833 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:27.833 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836556 2024-02-05T06:53:27.834 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:27.835 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:27.836 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836556 2024-02-05T06:53:27.836 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:27.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836556 2024-02-05T06:53:27.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-02-05T06:53:27.838 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836556 2024-02-05T06:53:27.838 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:28.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836556 -lt 21474836556 2024-02-05T06:53:28.138 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:28.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:28.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673034 2024-02-05T06:53:28.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:28.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673034 2024-02-05T06:53:28.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:28.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673034 2024-02-05T06:53:28.143 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673034 2024-02-05T06:53:28.144 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673034' 2024-02-05T06:53:28.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:28.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673034 -lt 42949673034 2024-02-05T06:53:28.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:28.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509511 2024-02-05T06:53:28.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:28.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:28.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509511 2024-02-05T06:53:28.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:28.472 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509511 2024-02-05T06:53:28.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509511 2024-02-05T06:53:28.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509511' 2024-02-05T06:53:28.472 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:28.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509510 -lt 64424509511 2024-02-05T06:53:28.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:29.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:29.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:30.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509512 -lt 64424509511 2024-02-05T06:53:30.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:30.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:30.093 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:30.388 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:30.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:30.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:30.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:30.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:30.404 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:30.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:30.698 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:30.698 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:30.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:30.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:30.698 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:30.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836558 2024-02-05T06:53:30.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836558 2024-02-05T06:53:30.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558' 2024-02-05T06:53:30.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:30.862 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:30.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673036 2024-02-05T06:53:30.994 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673036 2024-02-05T06:53:30.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673036' 2024-02-05T06:53:30.995 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:30.995 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:31.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509513 2024-02-05T06:53:31.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509513 2024-02-05T06:53:31.131 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673036 2-64424509513' 2024-02-05T06:53:31.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:31.132 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836558 2024-02-05T06:53:31.132 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:31.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:31.134 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836558 2024-02-05T06:53:31.134 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:31.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836558 2024-02-05T06:53:31.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2024-02-05T06:53:31.135 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836558 2024-02-05T06:53:31.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:31.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836557 -lt 21474836558 2024-02-05T06:53:31.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:32.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:32.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:32.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836558 -lt 21474836558 2024-02-05T06:53:32.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:32.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673036 2024-02-05T06:53:32.778 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:32.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:32.781 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673036 2024-02-05T06:53:32.781 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:32.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673036 2024-02-05T06:53:32.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673036' 2024-02-05T06:53:32.782 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673036 2024-02-05T06:53:32.783 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:33.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673036 -lt 42949673036 2024-02-05T06:53:33.121 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:33.122 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509513 2024-02-05T06:53:33.122 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:33.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:33.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509513 2024-02-05T06:53:33.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509513 2024-02-05T06:53:33.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509513' 2024-02-05T06:53:33.127 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509513 2024-02-05T06:53:33.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:33.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509513 -lt 64424509513 2024-02-05T06:53:33.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:33.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:33.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:33.729 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:33.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:33.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:33.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:33.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:33.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:34.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:34.096 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:34.096 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:34.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:34.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:34.097 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:34.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836560 2024-02-05T06:53:34.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836560 2024-02-05T06:53:34.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560' 2024-02-05T06:53:34.257 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:34.257 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:34.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673037 2024-02-05T06:53:34.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673037 2024-02-05T06:53:34.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673037' 2024-02-05T06:53:34.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:34.393 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:34.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509515 2024-02-05T06:53:34.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509515 2024-02-05T06:53:34.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673037 2-64424509515' 2024-02-05T06:53:34.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:34.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836560 2024-02-05T06:53:34.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:34.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:34.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836560 2024-02-05T06:53:34.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:34.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836560 2024-02-05T06:53:34.534 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836560 2024-02-05T06:53:34.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836560' 2024-02-05T06:53:34.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:34.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836558 -lt 21474836560 2024-02-05T06:53:34.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:35.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:35.825 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:36.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836560 -lt 21474836560 2024-02-05T06:53:36.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:36.137 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673037 2024-02-05T06:53:36.137 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:36.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:36.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673037 2024-02-05T06:53:36.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:36.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673037 2024-02-05T06:53:36.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673037' 2024-02-05T06:53:36.141 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673037 2024-02-05T06:53:36.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:36.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673038 -lt 42949673037 2024-02-05T06:53:36.453 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:36.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509515 2024-02-05T06:53:36.455 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:36.456 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:36.456 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509515 2024-02-05T06:53:36.457 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:36.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509515 2024-02-05T06:53:36.458 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509515' 2024-02-05T06:53:36.458 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509515 2024-02-05T06:53:36.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:36.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509515 -lt 64424509515 2024-02-05T06:53:36.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:36.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:36.771 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:37.083 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:37.100 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:37.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:37.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:37.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:37.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:37.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:37.384 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:37.384 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:37.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:37.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:37.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:37.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836561 2024-02-05T06:53:37.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836561 2024-02-05T06:53:37.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561' 2024-02-05T06:53:37.554 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:37.554 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:37.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673039 2024-02-05T06:53:37.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673039 2024-02-05T06:53:37.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673039' 2024-02-05T06:53:37.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:37.702 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:37.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509516 2024-02-05T06:53:37.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509516 2024-02-05T06:53:37.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673039 2-64424509516' 2024-02-05T06:53:37.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:37.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836561 2024-02-05T06:53:37.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:37.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:37.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836561 2024-02-05T06:53:37.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:37.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836561 2024-02-05T06:53:37.844 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836561 2024-02-05T06:53:37.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2024-02-05T06:53:37.845 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:38.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836561 -lt 21474836561 2024-02-05T06:53:38.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:38.130 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673039 2024-02-05T06:53:38.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:38.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:38.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673039 2024-02-05T06:53:38.133 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:38.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673039 2024-02-05T06:53:38.134 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673039 2024-02-05T06:53:38.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673039' 2024-02-05T06:53:38.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:38.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673039 -lt 42949673039 2024-02-05T06:53:38.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:38.456 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:38.456 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509516 2024-02-05T06:53:38.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:38.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509516 2024-02-05T06:53:38.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:38.460 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509516 2024-02-05T06:53:38.460 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509516 2024-02-05T06:53:38.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509516' 2024-02-05T06:53:38.461 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:38.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509515 -lt 64424509516 2024-02-05T06:53:38.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:39.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:39.811 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:40.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509517 -lt 64424509516 2024-02-05T06:53:40.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:40.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:40.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:40.406 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:40.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:40.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:40.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:40.423 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:40.424 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:40.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:40.740 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:40.740 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:40.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:40.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:40.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:40.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836563 2024-02-05T06:53:40.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836563 2024-02-05T06:53:40.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563' 2024-02-05T06:53:40.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:40.910 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:41.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673041 2024-02-05T06:53:41.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673041 2024-02-05T06:53:41.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673041' 2024-02-05T06:53:41.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:41.051 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:41.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509518 2024-02-05T06:53:41.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509518 2024-02-05T06:53:41.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673041 2-64424509518' 2024-02-05T06:53:41.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:41.176 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836563 2024-02-05T06:53:41.176 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:41.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:41.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836563 2024-02-05T06:53:41.177 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:41.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836563 2024-02-05T06:53:41.178 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836563 2024-02-05T06:53:41.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836563' 2024-02-05T06:53:41.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:41.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836562 -lt 21474836563 2024-02-05T06:53:41.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:42.457 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:42.458 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:42.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836563 -lt 21474836563 2024-02-05T06:53:42.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:42.766 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:42.766 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673041 2024-02-05T06:53:42.767 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:42.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673041 2024-02-05T06:53:42.768 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:42.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673041 2024-02-05T06:53:42.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673041' 2024-02-05T06:53:42.770 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673041 2024-02-05T06:53:42.770 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:43.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673041 -lt 42949673041 2024-02-05T06:53:43.092 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:43.093 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509518 2024-02-05T06:53:43.093 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:43.094 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:43.095 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509518 2024-02-05T06:53:43.096 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:43.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509518 2024-02-05T06:53:43.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-02-05T06:53:43.098 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509518 2024-02-05T06:53:43.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:43.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509518 -lt 64424509518 2024-02-05T06:53:43.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:43.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:43.437 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:43.792 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:43.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:43.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:43.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:43.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:43.810 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:44.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:44.105 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:44.105 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:44.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:44.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:44.106 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:44.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836565 2024-02-05T06:53:44.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836565 2024-02-05T06:53:44.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565' 2024-02-05T06:53:44.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:44.289 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:44.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673042 2024-02-05T06:53:44.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673042 2024-02-05T06:53:44.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673042' 2024-02-05T06:53:44.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:44.431 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:44.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509520 2024-02-05T06:53:44.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509520 2024-02-05T06:53:44.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673042 2-64424509520' 2024-02-05T06:53:44.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:44.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836565 2024-02-05T06:53:44.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:44.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:44.605 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836565 2024-02-05T06:53:44.605 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:44.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836565 2024-02-05T06:53:44.607 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836565 2024-02-05T06:53:44.608 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-02-05T06:53:44.608 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:44.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836563 -lt 21474836565 2024-02-05T06:53:44.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:45.897 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:45.899 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:46.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836565 -lt 21474836565 2024-02-05T06:53:46.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:46.200 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673042 2024-02-05T06:53:46.200 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:46.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:46.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673042 2024-02-05T06:53:46.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:46.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673042 2024-02-05T06:53:46.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673042' 2024-02-05T06:53:46.204 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673042 2024-02-05T06:53:46.204 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:46.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673043 -lt 42949673042 2024-02-05T06:53:46.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:46.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509520 2024-02-05T06:53:46.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:46.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:46.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509520 2024-02-05T06:53:46.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:46.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509520 2024-02-05T06:53:46.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-02-05T06:53:46.525 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509520 2024-02-05T06:53:46.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:46.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509520 -lt 64424509520 2024-02-05T06:53:46.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:46.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:46.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:47.089 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:47.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:47.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:47.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:47.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:47.108 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:47.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:47.397 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:47.397 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:47.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:47.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:47.398 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:47.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836566 2024-02-05T06:53:47.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836566 2024-02-05T06:53:47.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566' 2024-02-05T06:53:47.568 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:47.569 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:47.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673044 2024-02-05T06:53:47.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673044 2024-02-05T06:53:47.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673044' 2024-02-05T06:53:47.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:47.704 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:47.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509521 2024-02-05T06:53:47.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509521 2024-02-05T06:53:47.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673044 2-64424509521' 2024-02-05T06:53:47.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:47.840 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836566 2024-02-05T06:53:47.840 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:47.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:47.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:47.841 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836566 2024-02-05T06:53:47.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836566 2024-02-05T06:53:47.842 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836566' 2024-02-05T06:53:47.843 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836566 2024-02-05T06:53:47.843 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:48.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836566 -lt 21474836566 2024-02-05T06:53:48.145 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:48.146 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673044 2024-02-05T06:53:48.146 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:48.147 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:48.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673044 2024-02-05T06:53:48.148 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:48.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673044 2024-02-05T06:53:48.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2024-02-05T06:53:48.149 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673044 2024-02-05T06:53:48.150 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:48.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673044 -lt 42949673044 2024-02-05T06:53:48.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:48.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509521 2024-02-05T06:53:48.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:48.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:48.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:48.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509521 2024-02-05T06:53:48.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509521 2024-02-05T06:53:48.486 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509521 2024-02-05T06:53:48.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-02-05T06:53:48.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:48.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509520 -lt 64424509521 2024-02-05T06:53:48.806 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:49.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:49.808 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:50.113 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509522 -lt 64424509521 2024-02-05T06:53:50.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:50.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:50.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:50.416 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:50.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:50.432 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:50.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:50.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:50.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:50.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:50.740 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:50.740 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:50.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:50.740 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:50.741 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:50.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836568 2024-02-05T06:53:50.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836568 2024-02-05T06:53:50.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568' 2024-02-05T06:53:50.895 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:50.895 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:51.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673046 2024-02-05T06:53:51.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673046 2024-02-05T06:53:51.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673046' 2024-02-05T06:53:51.031 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:51.032 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:51.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509523 2024-02-05T06:53:51.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509523 2024-02-05T06:53:51.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673046 2-64424509523' 2024-02-05T06:53:51.170 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:51.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836568 2024-02-05T06:53:51.171 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:51.172 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:51.173 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836568 2024-02-05T06:53:51.173 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:51.174 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836568 2024-02-05T06:53:51.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836568 2024-02-05T06:53:51.175 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-02-05T06:53:51.175 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:51.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836567 -lt 21474836568 2024-02-05T06:53:51.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:52.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:52.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:52.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836568 -lt 21474836568 2024-02-05T06:53:52.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:52.787 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:52.788 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673046 2024-02-05T06:53:52.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:52.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:52.790 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673046 2024-02-05T06:53:52.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673046 2024-02-05T06:53:52.791 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-02-05T06:53:52.791 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673046 2024-02-05T06:53:52.792 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:53.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673046 -lt 42949673046 2024-02-05T06:53:53.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:53.097 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509523 2024-02-05T06:53:53.097 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:53.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:53.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509523 2024-02-05T06:53:53.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:53.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509523 2024-02-05T06:53:53.101 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509523 2024-02-05T06:53:53.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509523' 2024-02-05T06:53:53.102 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:53.418 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509523 -lt 64424509523 2024-02-05T06:53:53.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:53.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:53.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:53.736 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:53.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:53.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:53.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:53.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:53.753 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:54.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:54.074 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:54.074 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:54.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:54.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:54.075 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:54.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836570 2024-02-05T06:53:54.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836570 2024-02-05T06:53:54.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570' 2024-02-05T06:53:54.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:54.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:54.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673047 2024-02-05T06:53:54.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673047 2024-02-05T06:53:54.378 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673047' 2024-02-05T06:53:54.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:54.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:54.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509525 2024-02-05T06:53:54.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509525 2024-02-05T06:53:54.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673047 2-64424509525' 2024-02-05T06:53:54.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:54.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836570 2024-02-05T06:53:54.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:54.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:54.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836570 2024-02-05T06:53:54.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:54.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836570 2024-02-05T06:53:54.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2024-02-05T06:53:54.536 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836570 2024-02-05T06:53:54.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:54.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836568 -lt 21474836570 2024-02-05T06:53:54.811 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:53:55.812 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:53:55.813 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:56.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836570 -lt 21474836570 2024-02-05T06:53:56.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:56.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673047 2024-02-05T06:53:56.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:56.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:56.128 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673047 2024-02-05T06:53:56.128 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:56.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673047 2024-02-05T06:53:56.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2024-02-05T06:53:56.129 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673047 2024-02-05T06:53:56.130 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:56.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673048 -lt 42949673047 2024-02-05T06:53:56.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:56.435 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509525 2024-02-05T06:53:56.435 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:56.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:56.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509525 2024-02-05T06:53:56.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:56.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509525 2024-02-05T06:53:56.438 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509525 2024-02-05T06:53:56.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-02-05T06:53:56.439 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:56.758 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509525 2024-02-05T06:53:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:56.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:57.056 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:57.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:57.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:57.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:57.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:57.072 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:57.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:57.363 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:57.363 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:57.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:57.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:57.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:57.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836571 2024-02-05T06:53:57.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836571 2024-02-05T06:53:57.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571' 2024-02-05T06:53:57.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:57.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:57.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673049 2024-02-05T06:53:57.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673049 2024-02-05T06:53:57.653 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673049' 2024-02-05T06:53:57.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:57.654 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:57.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509526 2024-02-05T06:53:57.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509526 2024-02-05T06:53:57.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673049 2-64424509526' 2024-02-05T06:53:57.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:57.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:57.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836571 2024-02-05T06:53:57.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:57.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836571 2024-02-05T06:53:57.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:57.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836571 2024-02-05T06:53:57.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-02-05T06:53:57.796 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836571 2024-02-05T06:53:57.797 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:53:58.096 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836571 -lt 21474836571 2024-02-05T06:53:58.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:58.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673049 2024-02-05T06:53:58.098 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:58.099 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:53:58.100 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673049 2024-02-05T06:53:58.101 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:58.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673049 2024-02-05T06:53:58.102 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673049 2024-02-05T06:53:58.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-02-05T06:53:58.102 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:53:58.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673049 -lt 42949673049 2024-02-05T06:53:58.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:58.420 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509526 2024-02-05T06:53:58.420 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:58.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:53:58.423 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509526 2024-02-05T06:53:58.423 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:58.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509526 2024-02-05T06:53:58.424 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509526 2024-02-05T06:53:58.425 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-02-05T06:53:58.425 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:53:58.752 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509526 -lt 64424509526 2024-02-05T06:53:58.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:53:58.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:53:58.753 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:53:59.060 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:53:59.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:53:59.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:53:59.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:53:59.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:53:59.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:53:59.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:53:59.385 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:53:59.385 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:53:59.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:53:59.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:59.386 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:53:59.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836573 2024-02-05T06:53:59.565 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836573 2024-02-05T06:53:59.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573' 2024-02-05T06:53:59.566 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:59.566 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:53:59.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673050 2024-02-05T06:53:59.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673050 2024-02-05T06:53:59.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673050' 2024-02-05T06:53:59.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:53:59.727 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:53:59.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509528 2024-02-05T06:53:59.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509528 2024-02-05T06:53:59.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509528' 2024-02-05T06:53:59.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:53:59.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836573 2024-02-05T06:53:59.861 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:53:59.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:53:59.862 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836573 2024-02-05T06:53:59.862 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:53:59.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836573 2024-02-05T06:53:59.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-02-05T06:53:59.863 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836573 2024-02-05T06:53:59.863 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:54:00.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836573 -lt 21474836573 2024-02-05T06:54:00.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:00.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673050 2024-02-05T06:54:00.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:00.180 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:54:00.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:00.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673050 2024-02-05T06:54:00.182 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673050 2024-02-05T06:54:00.182 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673050 2024-02-05T06:54:00.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-02-05T06:54:00.183 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:54:00.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673050 -lt 42949673050 2024-02-05T06:54:00.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:00.527 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509528 2024-02-05T06:54:00.527 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:00.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:54:00.529 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509528 2024-02-05T06:54:00.529 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:00.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509528 2024-02-05T06:54:00.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509528' 2024-02-05T06:54:00.531 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509528 2024-02-05T06:54:00.531 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:54:00.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509527 -lt 64424509528 2024-02-05T06:54:00.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:54:01.886 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:54:01.887 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:54:02.196 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509528 -lt 64424509528 2024-02-05T06:54:02.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:54:02.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:54:02.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:54:02.499 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:54:02.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:54:02.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:54:02.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:54:02.516 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:54:02.517 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:54:02.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:54:02.822 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:54:02.822 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:54:02.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:54:02.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:02.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:54:02.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836574 2024-02-05T06:54:02.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836574 2024-02-05T06:54:02.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574' 2024-02-05T06:54:02.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:02.982 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:54:03.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673052 2024-02-05T06:54:03.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673052 2024-02-05T06:54:03.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673052' 2024-02-05T06:54:03.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:03.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:54:03.258 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509530 2024-02-05T06:54:03.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509530 2024-02-05T06:54:03.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673052 2-64424509530' 2024-02-05T06:54:03.259 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:03.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836574 2024-02-05T06:54:03.259 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:03.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:54:03.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836574 2024-02-05T06:54:03.261 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:03.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836574 2024-02-05T06:54:03.262 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836574 2024-02-05T06:54:03.262 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-02-05T06:54:03.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:54:03.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836573 -lt 21474836574 2024-02-05T06:54:03.546 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:54:04.547 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:54:04.551 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:54:04.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836574 -lt 21474836574 2024-02-05T06:54:04.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:04.861 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673052 2024-02-05T06:54:04.862 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:04.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:54:04.864 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673052 2024-02-05T06:54:04.864 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:04.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673052 2024-02-05T06:54:04.865 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673052 2024-02-05T06:54:04.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-02-05T06:54:04.866 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:54:05.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673052 2024-02-05T06:54:05.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:05.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509530 2024-02-05T06:54:05.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:05.204 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:54:05.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509530 2024-02-05T06:54:05.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:05.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509530 2024-02-05T06:54:05.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-02-05T06:54:05.207 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509530 2024-02-05T06:54:05.207 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:54:05.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509530 -lt 64424509530 2024-02-05T06:54:05.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:54:05.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:54:05.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:54:05.872 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:54:05.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:54:05.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:54:05.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:54:05.890 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:54:05.891 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:54:06.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:54:06.174 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:54:06.174 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:54:06.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:54:06.174 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:06.175 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:54:06.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836576 2024-02-05T06:54:06.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836576 2024-02-05T06:54:06.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576' 2024-02-05T06:54:06.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:06.344 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:54:06.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673054 2024-02-05T06:54:06.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673054 2024-02-05T06:54:06.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673054' 2024-02-05T06:54:06.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:06.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:54:06.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509531 2024-02-05T06:54:06.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509531 2024-02-05T06:54:06.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673054 2-64424509531' 2024-02-05T06:54:06.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:06.605 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836576 2024-02-05T06:54:06.605 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:06.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:54:06.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836576 2024-02-05T06:54:06.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:06.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836576 2024-02-05T06:54:06.607 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836576 2024-02-05T06:54:06.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2024-02-05T06:54:06.607 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:54:06.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836575 -lt 21474836576 2024-02-05T06:54:06.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:54:07.885 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:54:07.886 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:54:08.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836576 2024-02-05T06:54:08.181 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:08.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673054 2024-02-05T06:54:08.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:08.183 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:54:08.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673054 2024-02-05T06:54:08.184 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:08.185 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673054 2024-02-05T06:54:08.186 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673054 2024-02-05T06:54:08.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-02-05T06:54:08.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:54:08.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673054 -lt 42949673054 2024-02-05T06:54:08.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:08.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509531 2024-02-05T06:54:08.523 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:08.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:54:08.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509531 2024-02-05T06:54:08.526 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:08.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509531 2024-02-05T06:54:08.527 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509531 2024-02-05T06:54:08.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-02-05T06:54:08.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:54:08.837 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509531 -lt 64424509531 2024-02-05T06:54:08.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:54:08.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:54:08.838 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:54:09.118 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:54:09.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:54:09.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:54:09.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:54:09.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:54:09.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:54:09.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:54:09.426 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:54:09.426 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:54:09.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:54:09.426 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:09.426 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:54:09.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836578 2024-02-05T06:54:09.598 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836578 2024-02-05T06:54:09.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578' 2024-02-05T06:54:09.599 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:09.599 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:54:09.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673055 2024-02-05T06:54:09.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673055 2024-02-05T06:54:09.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673055' 2024-02-05T06:54:09.725 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:09.725 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:54:09.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509533 2024-02-05T06:54:09.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509533 2024-02-05T06:54:09.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673055 2-64424509533' 2024-02-05T06:54:09.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:09.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836578 2024-02-05T06:54:09.860 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:09.860 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:54:09.862 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836578 2024-02-05T06:54:09.862 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:09.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836578 2024-02-05T06:54:09.862 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836578 2024-02-05T06:54:09.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2024-02-05T06:54:09.863 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:54:10.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836578 -lt 21474836578 2024-02-05T06:54:10.148 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:10.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673055 2024-02-05T06:54:10.149 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:10.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:54:10.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673055 2024-02-05T06:54:10.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:10.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673055 2024-02-05T06:54:10.154 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673055 2024-02-05T06:54:10.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-02-05T06:54:10.154 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:54:10.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673055 -lt 42949673055 2024-02-05T06:54:10.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:10.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509533 2024-02-05T06:54:10.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:10.475 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:54:10.476 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509533 2024-02-05T06:54:10.476 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:10.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509533 2024-02-05T06:54:10.477 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509533 2024-02-05T06:54:10.477 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-02-05T06:54:10.478 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:54:10.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509532 -lt 64424509533 2024-02-05T06:54:10.779 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:54:11.781 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:54:11.782 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:54:12.097 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509533 -lt 64424509533 2024-02-05T06:54:12.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:54:12.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:54:12.098 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:54:12.403 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:54:12.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:54:12.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:54:12.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:54:12.420 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:54:12.421 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:54:12.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:54:12.768 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:54:12.768 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:54:12.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:54:12.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:12.769 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:54:12.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836579 2024-02-05T06:54:12.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836579 2024-02-05T06:54:12.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579' 2024-02-05T06:54:12.933 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:12.933 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:54:13.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673057 2024-02-05T06:54:13.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673057 2024-02-05T06:54:13.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673057' 2024-02-05T06:54:13.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:13.072 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:54:13.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509535 2024-02-05T06:54:13.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509535 2024-02-05T06:54:13.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673057 2-64424509535' 2024-02-05T06:54:13.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:13.200 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836579 2024-02-05T06:54:13.200 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:13.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:54:13.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836579 2024-02-05T06:54:13.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:13.202 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836579 2024-02-05T06:54:13.203 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836579 2024-02-05T06:54:13.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-02-05T06:54:13.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:54:13.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836578 -lt 21474836579 2024-02-05T06:54:13.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:54:14.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:54:14.509 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:54:14.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836579 2024-02-05T06:54:14.816 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:14.817 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673057 2024-02-05T06:54:14.817 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:14.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:54:14.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:14.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673057 2024-02-05T06:54:14.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673057 2024-02-05T06:54:14.821 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673057 2024-02-05T06:54:14.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-02-05T06:54:14.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:54:15.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673057 -lt 42949673057 2024-02-05T06:54:15.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:15.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509535 2024-02-05T06:54:15.158 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:15.160 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:54:15.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:15.161 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509535 2024-02-05T06:54:15.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509535 2024-02-05T06:54:15.162 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509535 2024-02-05T06:54:15.163 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509535' 2024-02-05T06:54:15.163 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:54:15.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509535 -lt 64424509535 2024-02-05T06:54:15.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:54:15.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:54:15.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:54:15.810 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:54:15.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:54:15.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:54:15.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:54:15.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:54:15.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:54:16.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:54:16.111 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:54:16.111 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:54:16.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:54:16.111 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:16.111 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:54:16.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836581 2024-02-05T06:54:16.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836581 2024-02-05T06:54:16.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581' 2024-02-05T06:54:16.250 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:16.250 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:54:16.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673059 2024-02-05T06:54:16.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673059 2024-02-05T06:54:16.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673059' 2024-02-05T06:54:16.383 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:54:16.383 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:54:16.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509536 2024-02-05T06:54:16.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509536 2024-02-05T06:54:16.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673059 2-64424509536' 2024-02-05T06:54:16.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:16.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836581 2024-02-05T06:54:16.533 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:16.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:54:16.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836581 2024-02-05T06:54:16.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:16.535 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836581 2024-02-05T06:54:16.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836581 2024-02-05T06:54:16.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-02-05T06:54:16.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:54:16.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836580 -lt 21474836581 2024-02-05T06:54:16.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:54:17.826 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:54:17.827 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:54:18.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836581 -lt 21474836581 2024-02-05T06:54:18.136 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:18.137 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673059 2024-02-05T06:54:18.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:18.140 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:54:18.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673059 2024-02-05T06:54:18.140 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:18.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673059 2024-02-05T06:54:18.141 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949673059 2024-02-05T06:54:18.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-02-05T06:54:18.142 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:54:18.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673059 -lt 42949673059 2024-02-05T06:54:18.441 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:54:18.442 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509536 2024-02-05T06:54:18.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:54:18.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:54:18.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509536 2024-02-05T06:54:18.445 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:54:18.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509536 2024-02-05T06:54:18.446 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509536 2024-02-05T06:54:18.447 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-02-05T06:54:18.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:54:18.763 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509536 -lt 64424509536 2024-02-05T06:54:18.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:54:18.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:54:18.764 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:54:19.087 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:54:19.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:383: _scrub_abort: break 2024-02-05T06:54:19.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:385: _scrub_abort: set +o pipefail 2024-02-05T06:54:19.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:387: _scrub_abort: sleep 5 2024-02-05T06:54:24.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:389: _scrub_abort: grep 'nodeep_scrub set, aborting' td/osd-scrub-test/osd.1.log 2024-02-05T06:54:24.125 INFO:tasks.workunit.client.0.smithi148.stdout:2024-02-05T06:52:20.088+0000 7fe677811640 10 osd.1 pg_epoch: 21 pg[1.0( v 19'1000 (0'0,19'1000] local-lis/les=17/18 n=1000 ec=17/17 lis/c=17/17 les/c/f=18/18/0 sis=17) [1,0,2] r=0 lpr=17 crt=19'1000 lcod 19'999 mlcod 19'999 active+clean+scrubbing+deep [ 1.0: ] TIME_FOR_DEEP] scrubber: nodeep_scrub set, aborting 2024-02-05T06:54:24.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:395: _scrub_abort: get_last_scrub_stamp 1.0 2024-02-05T06:54:24.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:24.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:24.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:24.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:24.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:395: _scrub_abort: local last_scrub=2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:24.424 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:396: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2024-02-05T06:54:24.844 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:398: _scrub_abort: ceph osd unset nodeep-scrub 2024-02-05T06:54:25.249 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is unset 2024-02-05T06:54:25.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:399: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2024-02-05T06:54:25.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:401: _scrub_abort: ceph osd unset noscrub 2024-02-05T06:54:25.776 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is unset 2024-02-05T06:54:25.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:403: _scrub_abort: TIMEOUT=500 2024-02-05T06:54:25.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:404: _scrub_abort: wait_for_scrub 1.0 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:25.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:54:25.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:25.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:54:25.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:54:25.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:25.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:25.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:25.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:25.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:25.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:26.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:26.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:27.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:27.089 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:27.089 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:27.089 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:27.090 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:27.090 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:27.090 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:27.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:27.384 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:28.385 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:28.386 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:28.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:28.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:28.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:28.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:28.387 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:28.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:28.686 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:29.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:29.688 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:29.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:29.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:29.689 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:29.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:29.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:29.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:29.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:30.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:30.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:30.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:30.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:30.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:30.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:30.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:31.282 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:31.283 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:32.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:32.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:32.286 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:32.286 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:32.286 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:32.286 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:32.286 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:32.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:32.591 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:33.592 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:33.593 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:33.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:33.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:33.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:33.594 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:33.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:33.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:33.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:34.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:34.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:34.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:34.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:34.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:34.908 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:34.908 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:35.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:35.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:36.200 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:36.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:36.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:36.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:36.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:36.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:36.203 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:36.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:36.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:37.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:37.505 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:37.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:37.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:37.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:37.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:37.506 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:37.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:37.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:38.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:38.810 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:38.811 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:38.811 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:38.811 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:38.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:38.812 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:39.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:39.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:40.124 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:40.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:40.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:40.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:40.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:40.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:40.126 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:40.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:40.440 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:41.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:41.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:41.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:41.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:41.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:41.443 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:41.444 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:41.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:41.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:42.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:42.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:42.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:42.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:42.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:42.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:42.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:43.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:43.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:44.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:44.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:44.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:44.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:44.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:44.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:44.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:44.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:44.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:45.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:45.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:45.382 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:45.382 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:45.382 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:45.382 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:45.382 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:45.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:45.697 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:46.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:46.698 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:46.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:46.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:46.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:46.699 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:46.700 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:47.012 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:48.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:48.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:48.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:48.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:48.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:48.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:48.015 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:48.332 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:48.333 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:49.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:49.334 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:49.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:49.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:49.335 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:49.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:49.336 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:49.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:49.648 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:50.649 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:50.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:50.650 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:50.650 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:50.650 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:50.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:50.651 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:50.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:50.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:51.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:51.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:51.975 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:51.975 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:51.975 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:51.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:51.976 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:52.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:52.278 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:53.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:53.279 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:53.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:53.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:53.280 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:53.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:53.281 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:53.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:53.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:54.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:54.579 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:54.580 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:54.580 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:54.580 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:54.581 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:54.581 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:54.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:54.871 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:55.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:55.873 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:55.874 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:55.874 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:55.874 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:55.875 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:55.875 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:56.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:56.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:57.187 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:57.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:57.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:57.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:57.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:57.189 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:57.190 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:57.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:57.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:58.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:58.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:58.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:58.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:58.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:58.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:58.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:54:58.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:54:58.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:54:59.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:54:59.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:54:59.793 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:54:59.793 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:54:59.793 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:54:59.794 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:54:59.794 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:00.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:00.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:01.102 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:01.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:01.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:01.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:01.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:01.105 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:01.105 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:01.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:01.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:02.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:02.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:02.415 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:02.415 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:02.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:02.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:02.416 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:02.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:02.726 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:03.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:03.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:03.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:03.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:03.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:03.729 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:03.730 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:04.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:04.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:05.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:05.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:05.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:05.041 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:05.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:05.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:05.042 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:05.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:05.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:06.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:06.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:06.348 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:06.348 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:06.348 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:06.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:06.349 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:06.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:06.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:07.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:07.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:07.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:07.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:07.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:07.662 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:07.662 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:07.965 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:07.966 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:08.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:08.967 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:08.968 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:08.968 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:08.968 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:08.969 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:08.969 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:09.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:09.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:10.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:10.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:10.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:10.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:10.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:10.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:10.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:10.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:10.573 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:11.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:11.574 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:11.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:11.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:11.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:11.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:11.575 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:11.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:11.874 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:12.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:12.875 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:12.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:12.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:12.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:12.877 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:12.878 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:13.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:13.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:14.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:14.179 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:14.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:14.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:14.181 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:14.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:14.182 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:14.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:14.479 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:15.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:15.480 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:15.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:15.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:15.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:15.481 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:15.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:15.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:15.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:16.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:16.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:16.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:16.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:16.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:16.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:16.791 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:17.082 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:17.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:18.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:18.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:18.085 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:18.085 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:18.085 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:18.086 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:18.086 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:18.393 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:18.394 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:19.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:19.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:19.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:19.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:19.396 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:19.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:19.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:19.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:19.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:20.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:20.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:20.712 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:20.712 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:20.712 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:20.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:20.713 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:21.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:21.023 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:22.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:22.024 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:22.025 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:22.025 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:22.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:22.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:22.026 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:22.327 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:22.328 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:23.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:23.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:23.330 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:23.331 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:23.331 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:23.331 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:23.331 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:23.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:23.633 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:24.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:24.634 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:24.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:24.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:24.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:24.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:24.636 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:24.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:24.932 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:25.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:25.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:25.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:25.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:25.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:25.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:25.935 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:26.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:26.235 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:27.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:27.237 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:27.237 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:27.237 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:27.238 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:27.238 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:27.238 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:27.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:27.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:28.537 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:28.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:28.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:28.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:28.538 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:28.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:28.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:28.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:28.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:29.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:29.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:29.852 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:29.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:29.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:29.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:29.853 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:30.153 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:30.154 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:31.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:31.155 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:31.156 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:31.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:31.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:31.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:31.157 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:31.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:31.450 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:32.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:32.452 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:32.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:32.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:32.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:32.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:32.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:32.792 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:32.793 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:33.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:33.794 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:33.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:33.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:33.795 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:33.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:33.796 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:34.106 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:34.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:35.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:35.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:35.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:35.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:35.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:35.109 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:35.110 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:35.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:35.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:36.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:36.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:36.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:36.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:36.414 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:36.415 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:36.415 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:36.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:36.716 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:37.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:37.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:37.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:37.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:37.718 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:37.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:37.719 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:38.013 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:38.014 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:39.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:39.015 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:39.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:39.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:39.016 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:39.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:39.017 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:39.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:39.321 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:40.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:40.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:40.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:40.324 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:40.325 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:40.325 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:40.325 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:40.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:40.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:41.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:41.640 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:41.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:41.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:41.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:41.642 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:41.642 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:41.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:41.956 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:42.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:42.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:42.958 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:42.958 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:42.958 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:42.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:42.959 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:43.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:43.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:44.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:44.271 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:44.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:44.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:44.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:44.273 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:44.274 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:44.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:44.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:45.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:45.583 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:45.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:45.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:45.584 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:45.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:45.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:45.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:45.898 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:46.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:46.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:46.901 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:46.901 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:46.901 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:46.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:46.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:47.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:47.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:48.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:48.212 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:48.212 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:48.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:48.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:48.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:48.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:48.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:48.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:49.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:49.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:49.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:49.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:49.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:49.526 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:49.526 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:49.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:49.841 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:50.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:50.843 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:50.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:50.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:50.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:50.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:50.844 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:51.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:51.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:52.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:52.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:52.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:52.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:52.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:52.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:52.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:52.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:52.434 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:53.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:53.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:53.436 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:53.436 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:53.436 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:53.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:53.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:53.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:53.743 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:54.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:54.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:54.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:54.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:54.746 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:54.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:54.747 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:55.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:55.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:56.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:56.051 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:56.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:56.052 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:56.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:56.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:56.053 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:56.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:56.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:57.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:57.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:57.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:57.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:57.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:57.365 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:57.365 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:57.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:57.658 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:58.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:58.659 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:58.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:58.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:58.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:58.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:58.662 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:55:58.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:55:58.977 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:55:59.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:55:59.978 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:55:59.979 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:55:59.979 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:55:59.979 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:55:59.980 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:55:59.980 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:00.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:00.291 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:01.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:01.292 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:01.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:01.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:01.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:01.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:01.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:01.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:01.602 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:02.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:02.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:02.605 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:02.605 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:02.605 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:02.605 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:02.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:02.909 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:02.911 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:03.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:03.910 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:03.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:03.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:03.911 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:03.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:03.912 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:04.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:04.223 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:05.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:05.225 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:05.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:05.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:05.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:05.227 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:05.227 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:05.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:05.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:06.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:06.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:06.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:06.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:06.539 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:06.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:06.540 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:06.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:06.845 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:07.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:07.846 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:07.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:07.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:07.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:07.848 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:07.849 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:08.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:08.161 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:09.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:09.162 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:09.164 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:09.164 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:09.164 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:09.164 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:09.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:09.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:09.470 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:10.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:10.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:10.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:10.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:10.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:10.473 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:10.474 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:10.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:10.796 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:11.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:11.797 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:11.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:11.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:11.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:11.798 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:11.799 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:12.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:12.101 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:13.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:13.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:13.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:13.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:13.104 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:13.105 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:13.105 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:13.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:13.395 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:14.396 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:14.397 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:14.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:14.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:14.397 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:14.398 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:14.398 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:14.706 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:14.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:15.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:15.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:15.708 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:15.708 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:15.708 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:15.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:15.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:16.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:16.018 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:17.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:17.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:17.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:17.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:17.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:17.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:17.021 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:17.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:17.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:18.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:18.318 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:18.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:18.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:18.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:18.319 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:18.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:18.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:18.615 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:19.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:19.616 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:19.617 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:19.617 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:19.618 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:19.618 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:19.618 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:19.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:19.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:20.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:20.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:20.926 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:20.926 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:20.926 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:20.926 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:20.926 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:21.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:21.219 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:22.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:22.220 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:22.221 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:22.221 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:22.221 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:22.221 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:22.222 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:22.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:22.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:23.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:23.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:23.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:23.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:23.535 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:23.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:23.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:23.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:23.848 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:24.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:24.849 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:24.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:24.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:24.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:24.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:24.852 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:25.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:25.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:26.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:26.166 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:26.167 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:26.167 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:26.167 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:26.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:26.168 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:26.481 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:26.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:27.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:27.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:27.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:27.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:27.484 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:27.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:27.485 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:27.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:27.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:28.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:28.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:28.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:28.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:28.806 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:28.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:28.807 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:29.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:29.122 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:30.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:30.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:30.124 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:30.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:30.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:30.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:30.125 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:30.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:30.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:31.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:31.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:31.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:31.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:31.437 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:31.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:31.438 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:31.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:31.746 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:32.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:32.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:32.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:32.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:32.748 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:32.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:32.749 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:33.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:33.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:34.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:34.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:34.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:34.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:34.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:34.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:34.046 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:34.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:34.359 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:35.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:35.360 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:35.361 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:35.361 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:35.361 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:35.362 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:35.362 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:35.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:35.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:36.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:36.677 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:36.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:36.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:36.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:36.678 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:36.679 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:36.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:36.983 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:37.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:37.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:37.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:37.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:37.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:37.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:37.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:38.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:38.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:39.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:39.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:39.291 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:39.291 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:39.291 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:39.291 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:39.291 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:39.586 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:39.587 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:40.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:40.588 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:40.590 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:40.590 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:40.590 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:40.590 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:40.590 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:40.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:40.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:41.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:41.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:41.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:41.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:41.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:41.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:41.902 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:42.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:42.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:43.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:43.201 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:43.201 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:43.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:43.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:43.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:43.202 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:43.508 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:44.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:44.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:44.511 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:44.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:44.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:44.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:44.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:44.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:44.813 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:45.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:45.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:45.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:45.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:45.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:45.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:45.817 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:46.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:46.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:47.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:47.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:47.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:47.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:47.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:47.135 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:47.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:47.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:47.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:48.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:48.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:48.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:48.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:48.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:48.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:48.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:48.728 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:49.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:49.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:49.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:49.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:49.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:49.732 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:49.732 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:50.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:50.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:51.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:51.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:51.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:51.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:51.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:51.048 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:51.048 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:51.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:51.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:52.357 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:52.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:52.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:52.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:52.358 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:52.359 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:52.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:52.668 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:53.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:53.669 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:53.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:53.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:53.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:53.671 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:53.672 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:53.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:53.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:54.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:54.986 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:54.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:54.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:54.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:54.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:54.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:55.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:55.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:56.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:56.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:56.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:56.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:56.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:56.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:56.303 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:56.603 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:57.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:57.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:57.605 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:57.605 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:57.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:57.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:57.606 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:57.917 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:57.918 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:56:58.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:56:58.919 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:56:58.920 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:56:58.920 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:56:58.920 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:56:58.921 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:56:58.921 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:56:59.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:56:59.226 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:00.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:00.227 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:00.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:00.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:00.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:00.229 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:00.229 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:00.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:00.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:01.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:01.524 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:01.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:01.524 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:01.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:01.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:01.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:01.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:01.839 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:02.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:02.840 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:02.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:02.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:02.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:02.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:02.842 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:03.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:03.149 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:04.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:04.151 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:04.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:04.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:04.152 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:04.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:04.153 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:04.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:04.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:05.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:05.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:05.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:05.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:05.466 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:05.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:05.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:05.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:05.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:06.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:06.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:06.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:06.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:06.776 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:06.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:06.777 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:07.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:07.078 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:08.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:08.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:08.080 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:08.080 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:08.080 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:08.080 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:08.081 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:08.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:08.381 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:09.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:09.382 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:09.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:09.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:09.384 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:09.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:09.385 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:09.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:09.687 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:10.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:10.689 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:10.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:10.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:10.690 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:10.691 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:10.691 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:11.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:11.003 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:12.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:12.005 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:12.005 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:12.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:12.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:12.006 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:12.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:12.317 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:13.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:13.319 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:13.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:13.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:13.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:13.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:13.320 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:13.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:13.636 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:14.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:14.637 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:14.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:14.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:14.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:14.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:14.639 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:14.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:14.969 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:15.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:15.970 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:15.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:15.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:15.972 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:15.973 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:15.973 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:16.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:16.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:17.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:17.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:17.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:17.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:17.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:17.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:17.272 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:17.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:17.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:18.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:18.584 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:18.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:18.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:18.585 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:18.586 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:18.586 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:18.900 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:18.901 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:19.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:19.902 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:19.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:19.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:19.903 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:19.904 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:19.904 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:20.209 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:20.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:21.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:21.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:21.212 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:21.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:21.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:21.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:21.213 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:21.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:21.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:22.512 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:22.512 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:22.513 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:22.513 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:22.513 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:22.513 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:22.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:22.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:23.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:23.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:23.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:23.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:23.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:23.821 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:23.821 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:24.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:24.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:25.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:25.134 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:25.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:25.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:25.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:25.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:25.136 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:25.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:25.445 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:26.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:26.446 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:26.447 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:26.448 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:26.448 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:26.448 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:26.448 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:26.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:26.757 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:27.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:27.759 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:27.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:27.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:27.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:27.760 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:27.761 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:28.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:28.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:29.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:29.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:29.076 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:29.076 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:29.076 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:29.076 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:29.076 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:29.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:29.376 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:30.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:30.377 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:30.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:30.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:30.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:30.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:30.379 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:30.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:30.693 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:31.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:31.694 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:31.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:31.695 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:31.696 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:31.696 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:31.696 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:32.004 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:32.005 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:33.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:33.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:33.007 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:33.007 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:33.007 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:33.008 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:33.008 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:33.323 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:33.324 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:34.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:34.325 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:34.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:34.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:34.326 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:34.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:34.327 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:34.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:34.638 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:35.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:35.639 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:35.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:35.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:35.640 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:35.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:35.641 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:35.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:35.947 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:36.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:36.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:36.949 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:36.950 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:36.950 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:36.950 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:36.950 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:37.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:37.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:38.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:38.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:38.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:38.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:38.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:38.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:38.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:38.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:38.558 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:39.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:39.559 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:39.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:39.560 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:39.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:39.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:39.561 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:39.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:39.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:40.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:40.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:40.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:40.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:40.867 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:40.868 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:40.868 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:41.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:41.177 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:42.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:42.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:42.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:42.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:42.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:42.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:42.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:42.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:42.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:43.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:43.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:43.491 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:43.491 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:43.491 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:43.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:43.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:43.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:43.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:44.820 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:44.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:44.821 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:44.821 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:44.821 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:44.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:44.822 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:45.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:45.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:46.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:46.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:46.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:46.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:46.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:46.127 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:46.128 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:46.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:46.429 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:47.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:47.430 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:47.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:47.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:47.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:47.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:47.433 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:47.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:47.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:48.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:48.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:48.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:48.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:48.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:48.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:48.734 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:49.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:49.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:50.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:50.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:50.048 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:50.049 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:50.049 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:50.049 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:50.049 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:50.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:50.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:51.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:51.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:51.363 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:51.363 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:51.363 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:51.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:51.364 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:51.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:51.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:52.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:52.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:52.676 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:52.676 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:52.676 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:52.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:52.677 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:52.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:52.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:53.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:53.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:53.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:53.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:53.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:53.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:54.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:54.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:54.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:55.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:55.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:55.304 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:55.304 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:55.304 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:55.305 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:55.305 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:55.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:50:34.413289+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:55.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:57:56.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:57:56.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:57:56.613 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:57:56.613 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:57:56.613 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:57:56.614 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:57:56.614 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:57:56.920 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:57:54.864425+0000 '>' 2024-01-22T06:50:34.413289+0000 2024-02-05T06:57:56.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:57:56.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:405: _scrub_abort: perf_counters td/osd-scrub-test 3 2024-02-05T06:57:56.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-02-05T06:57:56.921 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-02-05T06:57:56.921 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-02-05T06:57:56.922 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-02-05T06:57:56.922 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:57:56.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-02-05T06:57:56.923 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T06:57:57.055 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:57:57.055 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_ec": [ 2024-02-05T06:57:57.055 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:57:57.056 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:57:57.056 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T06:57:57.056 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T06:57:57.056 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.056 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:57:57.056 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:57:57.056 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:57:57.056 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:57:57.056 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:57:57.056 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.056 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.056 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.056 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.057 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:57:57.057 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:57:57.057 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.057 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.057 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.057 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.057 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:57:57.057 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:57:57.057 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:57:57.057 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:57:57.057 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:57:57.057 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:57:57.058 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:57:57.058 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.058 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.058 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.058 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.058 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:57:57.058 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:57:57.058 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:57:57.058 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:57:57.058 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:57:57.058 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.058 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.058 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.058 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.059 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:57:57.059 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.059 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.059 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:57:57.059 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_repl": [ 2024-02-05T06:57:57.059 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:57:57.059 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:57:57.059 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T06:57:57.059 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T06:57:57.059 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.059 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:57:57.059 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:57:57.060 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:57:57.060 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:57:57.060 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:57:57.060 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.060 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.060 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.060 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.060 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:57:57.060 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:57:57.060 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.060 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.060 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.060 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.060 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:57:57.061 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:57:57.062 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.062 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.062 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.062 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.062 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:57:57.062 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.062 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.062 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:57:57.062 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_ec": [ 2024-02-05T06:57:57.062 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:57:57.062 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:57:57.062 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T06:57:57.062 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T06:57:57.062 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.063 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:57:57.063 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:57:57.063 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:57:57.063 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:57:57.063 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:57:57.063 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.063 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.063 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.063 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.063 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:57:57.063 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:57:57.063 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.063 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.063 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.064 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.064 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:57:57.064 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:57:57.064 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:57:57.064 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:57:57.064 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:57:57.064 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:57:57.064 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:57:57.064 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.064 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.064 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.064 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.064 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:57:57.064 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_repl": [ 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T06:57:57.065 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.066 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.067 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:57:57.067 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:57:57.067 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:57:57.067 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:57:57.068 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:57:57.068 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:57:57.068 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:57:57.068 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.068 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.069 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:57:57.070 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:57:57.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:57:57.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-02-05T06:57:57.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T06:57:57.196 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:57:57.197 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_ec": [ 2024-02-05T06:57:57.197 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:57:57.197 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:57:57.197 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T06:57:57.197 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T06:57:57.197 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.197 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:57:57.197 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:57:57.197 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:57:57.197 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:57:57.197 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:57:57.197 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.197 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.197 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:57:57.198 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.199 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_repl": [ 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 2, 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 2, 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 1, 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 1, 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 207.206848401, 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 207.206848401 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.200 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 1, 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 1, 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 5.003884683, 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 5.003884683 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 41, 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 2, 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 2, 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0.051998684, 2024-02-05T06:57:57.201 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0.025999342 2024-02-05T06:57:57.202 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.202 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:57:57.202 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:57:57.202 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:57:57.202 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:57:57.202 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:57:57.202 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.202 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.202 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.202 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.202 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 2 2024-02-05T06:57:57.202 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.202 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.202 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_ec": [ 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.203 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:57:57.204 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:57:57.204 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.204 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.204 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.204 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.204 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:57:57.204 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:57:57.204 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:57:57.204 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:57:57.204 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:57:57.204 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:57:57.204 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:57:57.204 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.204 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.205 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.205 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.205 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:57:57.205 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:57:57.205 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:57:57.205 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:57:57.205 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:57:57.205 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.205 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.205 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.205 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.205 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:57:57.205 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.205 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_repl": [ 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.206 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.207 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.207 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:57:57.207 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:57:57.207 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.207 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.207 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.207 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.207 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:57:57.207 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:57:57.209 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:57:57.209 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:57:57.209 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:57:57.209 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.210 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:57:57.211 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.211 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.211 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:57:57.211 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:57:57.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:57:57.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-02-05T06:57:57.211 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T06:57:57.337 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:57:57.337 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_ec": [ 2024-02-05T06:57:57.337 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:57:57.337 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:57:57.337 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T06:57:57.337 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T06:57:57.337 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.337 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:57:57.337 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:57:57.337 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:57:57.337 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:57:57.338 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:57:57.338 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.338 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.338 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.338 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.338 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:57:57.338 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:57:57.338 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.338 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.338 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.338 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.338 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:57:57.338 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:57:57.339 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:57:57.339 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:57:57.339 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:57:57.339 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:57:57.339 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:57:57.339 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.339 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.339 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.339 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.339 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:57:57.339 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:57:57.339 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:57:57.339 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:57:57.340 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:57:57.340 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.340 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.340 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.340 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.340 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:57:57.340 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.340 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.340 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:57:57.340 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_repl": [ 2024-02-05T06:57:57.340 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:57:57.340 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:57:57.341 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T06:57:57.341 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T06:57:57.341 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.341 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:57:57.341 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:57:57.341 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:57:57.341 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:57:57.341 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:57:57.341 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.341 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.341 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.341 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.341 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:57:57.341 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:57:57.342 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.342 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.342 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.342 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.342 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:57:57.342 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:57:57.342 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:57:57.342 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:57:57.342 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:57:57.342 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:57:57.342 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:57:57.342 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.342 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.343 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:57:57.344 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_ec": [ 2024-02-05T06:57:57.344 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:57:57.344 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:57:57.344 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T06:57:57.344 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T06:57:57.344 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.344 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:57:57.344 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:57:57.344 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:57:57.344 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:57:57.344 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:57:57.345 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.345 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.345 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.345 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.345 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:57:57.345 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:57:57.345 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.345 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.345 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.345 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.345 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:57:57.345 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:57:57.345 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:57:57.345 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:57:57.346 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:57:57.346 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:57:57.346 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:57:57.346 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.346 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.346 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.346 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.346 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:57:57.346 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:57:57.346 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:57:57.346 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:57:57.346 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:57:57.346 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.347 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.347 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.347 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.347 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:57:57.347 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.347 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.347 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:57:57.347 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_repl": [ 2024-02-05T06:57:57.347 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:57:57.347 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:57:57.347 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T06:57:57.347 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T06:57:57.347 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.347 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:57:57.348 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:57:57.348 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:57:57.348 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:57:57.348 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:57:57.348 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.348 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.348 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.348 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.348 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:57:57.348 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:57:57.348 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.348 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.348 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.349 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.349 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:57:57.349 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:57:57.349 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:57:57.349 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:57:57.349 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:57:57.349 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:57:57.349 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:57:57.350 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.350 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.350 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.350 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.350 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:57:57.350 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:57:57.350 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:57:57.350 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:57:57.350 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:57:57.350 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:57:57.350 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:57:57.350 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:57:57.350 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:57:57.351 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:57:57.351 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.351 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:57:57.351 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:57:57.351 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:57:57.351 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_deep_scrub_abort ------------------' 2024-02-05T06:57:57.352 INFO:tasks.workunit.client.0.smithi148.stdout:-------------- Teardown Test TEST_deep_scrub_abort ------------------ 2024-02-05T06:57:57.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-02-05T06:57:57.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T06:57:57.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:57:57.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T06:57:57.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:57:57.353 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:57:57.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:57:57.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:57:57.353 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:57:57.459 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:57:57.460 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:57:57.461 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:57:57.462 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:57:57.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:57:57.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:57:57.463 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:57:57.463 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:57:57.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:57:57.464 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:57:57.464 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:57:57.465 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:57:57.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:57:57.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T06:57:57.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:57:57.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:57:57.522 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:57:57.522 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.545011 2024-02-05T06:57:57.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:57:57.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:57:57.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_deep_scrub_abort ------------------' 2024-02-05T06:57:57.523 INFO:tasks.workunit.client.0.smithi148.stdout:-------------- Complete Test TEST_deep_scrub_abort ------------------ 2024-02-05T06:57:57.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-02-05T06:57:57.523 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_dump_scrub_schedule -------------------' 2024-02-05T06:57:57.524 INFO:tasks.workunit.client.0.smithi148.stdout:-------------- Prepare Test TEST_dump_scrub_schedule ------------------- 2024-02-05T06:57:57.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-02-05T06:57:57.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-02-05T06:57:57.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-02-05T06:57:57.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T06:57:57.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:57:57.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T06:57:57.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:57:57.525 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:57:57.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:57:57.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:57:57.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:57:57.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:57:57.526 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:57:57.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:57:57.526 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:57:57.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:57:57.527 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:57:57.528 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:57:57.528 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:57:57.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:57:57.529 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:57:57.529 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:57:57.530 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:57:57.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:57:57.531 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T06:57:57.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:57:57.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:57:57.532 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:57:57.532 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.545011 2024-02-05T06:57:57.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:57:57.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:57:57.533 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-02-05T06:57:57.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:57:57.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:57:57.534 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:57:57.534 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.545011 2024-02-05T06:57:57.536 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:57:57.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:57:57.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:57:57.536 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-02-05T06:57:57.536 INFO:tasks.workunit.client.0.smithi148.stdout:-------------- Run Test TEST_dump_scrub_schedule ----------------------- 2024-02-05T06:57:57.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_dump_scrub_schedule -----------------------' 2024-02-05T06:57:57.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_dump_scrub_schedule td/osd-scrub-test 2024-02-05T06:57:57.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:526: TEST_dump_scrub_schedule: local dir=td/osd-scrub-test 2024-02-05T06:57:57.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:527: TEST_dump_scrub_schedule: local poolname=test 2024-02-05T06:57:57.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:528: TEST_dump_scrub_schedule: local OSDS=3 2024-02-05T06:57:57.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:529: TEST_dump_scrub_schedule: local objects=15 2024-02-05T06:57:57.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:531: TEST_dump_scrub_schedule: TESTDATA=testdata.545011 2024-02-05T06:57:57.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:533: TEST_dump_scrub_schedule: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-02-05T06:57:57.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-02-05T06:57:57.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:57:57.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:57:57.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:57:57.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-02-05T06:57:57.538 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-02-05T06:57:57.925 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:57:57.925 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:57:57.925 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:57:57.926 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:57:57.926 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:57:57.926 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:57:57.926 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:57:57.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-02-05T06:57:58.129 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:57:58.130 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:57:58.130 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:57:58.130 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:57:58.130 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:57:58.131 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:57:58.131 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:57:58.131 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:57:58.131 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:57:58.131 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:57:58.131 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:57:58.131 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:57:58.131 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.545011/ceph-mon.a.asok 2024-02-05T06:57:58.132 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:57:58.132 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.545011/ceph-mon.a.asok config get fsid 2024-02-05T06:57:58.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:57:58.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:57:58.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:57:58.242 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:57:58.243 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:57:58.243 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:57:58.243 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:57:58.243 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:57:58.244 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:57:58.244 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:57:58.244 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:57:58.244 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.545011/ceph-mon.a.asok 2024-02-05T06:57:58.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:57:58.245 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.545011/ceph-mon.a.asok config get mon_host 2024-02-05T06:57:58.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:534: TEST_dump_scrub_schedule: run_mgr td/osd-scrub-test x 2024-02-05T06:57:58.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-02-05T06:57:58.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:57:58.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:57:58.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:57:58.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-02-05T06:57:58.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:57:59.204 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:57:59.204 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:57:59.204 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:57:59.204 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:57:59.204 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:57:59.204 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:57:59.204 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:57:59.205 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:57:59.206 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:57:59.236 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:545: TEST_dump_scrub_schedule: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2' 2024-02-05T06:57:59.237 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: expr 3 - 1 2024-02-05T06:57:59.239 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: seq 0 2 2024-02-05T06:57:59.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:57:59.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 0 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-02-05T06:57:59.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:57:59.242 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:57:59.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:57:59.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:57:59.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-02-05T06:57:59.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:57:59.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:57:59.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:57:59.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:57:59.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-02-05T06:57:59.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-02-05T06:57:59.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:57:59.243 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:57:59.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:57:59.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:57:59.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:57:59.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:57:59.244 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:57:59.244 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:57:59.244 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:57:59.244 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:57:59.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:57:59.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:57:59.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:57:59.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:57:59.244 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:57:59.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:57:59.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:57:59.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:57:59.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:57:59.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:57:59.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:57:59.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2' 2024-02-05T06:57:59.245 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-02-05T06:57:59.246 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:57:59.246 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0774835d-1c1c-4d9b-845b-c2b05f284dc7 2024-02-05T06:57:59.246 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 0774835d-1c1c-4d9b-845b-c2b05f284dc7 2024-02-05T06:57:59.247 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0774835d-1c1c-4d9b-845b-c2b05f284dc7' 2024-02-05T06:57:59.247 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:57:59.260 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD3hsBlRYlPDxAAFBKqSojHT2r3tilw8dLtYA== 2024-02-05T06:57:59.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD3hsBlRYlPDxAAFBKqSojHT2r3tilw8dLtYA=="}' 2024-02-05T06:57:59.261 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0774835d-1c1c-4d9b-845b-c2b05f284dc7 -i td/osd-scrub-test/0/new.json 2024-02-05T06:57:59.507 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:57:59.525 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-02-05T06:57:59.526 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 --mkfs --key AQD3hsBlRYlPDxAAFBKqSojHT2r3tilw8dLtYA== --osd-uuid 0774835d-1c1c-4d9b-845b-c2b05f284dc7 2024-02-05T06:57:59.564 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:57:59.556+0000 7fe629e6a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:57:59.567 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:57:59.560+0000 7fe629e6a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:57:59.568 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:57:59.560+0000 7fe629e6a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:57:59.568 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:57:59.560+0000 7fe629e6a640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:57:59.569 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:57:59.560+0000 7fe629e6a640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:57:59.569 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:57:59.560+0000 7fe629e6a640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:57:59.642 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:57:59.636+0000 7fe629e6a640 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-02-05T06:58:05.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-02-05T06:58:05.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:58:05.741 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:58:05.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:58:05.741 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:58:06.125 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:58:06.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:58:06.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-02-05T06:58:06.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:58:06.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:58:06.127 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:58:06.161 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:06.156+0000 7fc61cefa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:06.208 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:06.200+0000 7fc61cefa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:06.209 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:06.204+0000 7fc61cefa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:06.412 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:58:06.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:58:06.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:58:06.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:58:06.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:58:06.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:58:06.413 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:58:06.413 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:58:06.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:58:06.414 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:58:06.736 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:58:06.787 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:06.780+0000 7fc61cefa640 -1 Falling back to public interface 2024-02-05T06:58:07.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:58:07.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:58:07.738 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:58:07.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:58:07.738 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:58:07.739 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:58:08.040 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:58:09.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:58:09.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:58:09.042 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:58:09.042 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:58:09.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:58:09.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:58:09.351 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:09.344+0000 7fc61cefa640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:58:09.361 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:58:10.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:58:10.362 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:58:10.363 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:58:10.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:58:10.363 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:58:10.364 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:58:10.704 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/49787445,v1:127.0.0.1:6803/49787445] [v2:127.0.0.1:6804/49787445,v1:127.0.0.1:6805/49787445] exists,up 0774835d-1c1c-4d9b-845b-c2b05f284dc7 2024-02-05T06:58:10.707 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:58:10.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:58:10.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:58:10.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:58:10.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 1 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-02-05T06:58:10.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:58:10.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:58:10.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:58:10.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:58:10.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-02-05T06:58:10.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:58:10.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:58:10.708 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:58:10.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:58:10.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-02-05T06:58:10.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-02-05T06:58:10.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:58:10.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:58:10.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:58:10.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:58:10.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:58:10.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:58:10.709 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:58:10.709 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:58:10.709 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:58:10.709 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:58:10.709 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:58:10.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:58:10.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:58:10.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:58:10.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:58:10.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:58:10.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:58:10.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:58:10.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:58:10.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:58:10.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:58:10.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2' 2024-02-05T06:58:10.710 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-02-05T06:58:10.711 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:58:10.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ab6779c1-9e74-4b95-b5c6-92512510744e 2024-02-05T06:58:10.711 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 ab6779c1-9e74-4b95-b5c6-92512510744e 2024-02-05T06:58:10.711 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ab6779c1-9e74-4b95-b5c6-92512510744e' 2024-02-05T06:58:10.711 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:58:10.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACh8Bl2OFAKxAAV/cWPS+y3QwoagKfVDVrHQ== 2024-02-05T06:58:10.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACh8Bl2OFAKxAAV/cWPS+y3QwoagKfVDVrHQ=="}' 2024-02-05T06:58:10.729 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ab6779c1-9e74-4b95-b5c6-92512510744e -i td/osd-scrub-test/1/new.json 2024-02-05T06:58:11.086 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:58:11.103 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-02-05T06:58:11.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 --mkfs --key AQACh8Bl2OFAKxAAV/cWPS+y3QwoagKfVDVrHQ== --osd-uuid ab6779c1-9e74-4b95-b5c6-92512510744e 2024-02-05T06:58:11.137 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:11.131+0000 7fa39ea2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:11.179 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:11.171+0000 7fa39ea2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:11.181 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:11.175+0000 7fa39ea2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:11.181 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:11.175+0000 7fa39ea2c640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:58:11.181 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:11.175+0000 7fa39ea2c640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:58:11.181 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:11.175+0000 7fa39ea2c640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:58:11.252 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:11.247+0000 7fa39ea2c640 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-02-05T06:58:17.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-02-05T06:58:17.717 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:58:17.719 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:58:17.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:58:17.719 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:58:18.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:58:18.107 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:58:18.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:58:18.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-02-05T06:58:18.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:58:18.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:58:18.142 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:18.135+0000 7fe3043f3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:18.168 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:18.159+0000 7fe3043f3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:18.169 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:18.163+0000 7fe3043f3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:18.442 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:58:18.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:58:18.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:58:18.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:58:18.443 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:58:18.443 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:58:18.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:58:18.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:58:18.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:58:18.444 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:58:18.738 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:18.731+0000 7fe3043f3640 -1 Falling back to public interface 2024-02-05T06:58:18.745 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:58:19.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:58:19.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:58:19.747 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:58:19.747 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:58:19.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:58:19.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:58:20.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:58:21.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:58:21.046 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:58:21.047 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:58:21.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:58:21.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:58:21.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:58:21.352 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:58:21.378 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:21.371+0000 7fe3043f3640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:58:22.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:58:22.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:58:22.354 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:58:22.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:58:22.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:58:22.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:58:22.698 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/205745906,v1:127.0.0.1:6811/205745906] [v2:127.0.0.1:6812/205745906,v1:127.0.0.1:6813/205745906] exists,up ab6779c1-9e74-4b95-b5c6-92512510744e 2024-02-05T06:58:22.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:58:22.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:58:22.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:58:22.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:58:22.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 2 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-02-05T06:58:22.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:58:22.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:58:22.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T06:58:22.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:58:22.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/2 2024-02-05T06:58:22.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:58:22.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:58:22.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:58:22.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:58:22.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-02-05T06:58:22.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-02-05T06:58:22.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:58:22.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:58:22.703 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:58:22.703 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:58:22.703 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:58:22.703 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:58:22.703 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:58:22.703 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:58:22.703 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:58:22.704 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:58:22.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:58:22.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:58:22.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:58:22.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:58:22.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:58:22.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:58:22.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:58:22.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:58:22.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:58:22.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:58:22.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:58:22.704 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2' 2024-02-05T06:58:22.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/2 2024-02-05T06:58:22.705 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:58:22.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6e9a52a2-fe29-4b8c-8720-e7350ef12e61 2024-02-05T06:58:22.705 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6e9a52a2-fe29-4b8c-8720-e7350ef12e61' 2024-02-05T06:58:22.706 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 6e9a52a2-fe29-4b8c-8720-e7350ef12e61 2024-02-05T06:58:22.706 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:58:22.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAOh8BlbkXqKhAATOyJ/al4AwwQhf2OzhSXlA== 2024-02-05T06:58:22.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAOh8BlbkXqKhAATOyJ/al4AwwQhf2OzhSXlA=="}' 2024-02-05T06:58:22.723 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6e9a52a2-fe29-4b8c-8720-e7350ef12e61 -i td/osd-scrub-test/2/new.json 2024-02-05T06:58:23.098 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:58:23.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/2/new.json 2024-02-05T06:58:23.117 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 --mkfs --key AQAOh8BlbkXqKhAATOyJ/al4AwwQhf2OzhSXlA== --osd-uuid 6e9a52a2-fe29-4b8c-8720-e7350ef12e61 2024-02-05T06:58:23.147 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:23.139+0000 7f9daae70640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:23.182 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:23.175+0000 7f9daae70640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:23.184 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:23.179+0000 7f9daae70640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:23.185 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:23.179+0000 7f9daae70640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:58:23.185 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:23.179+0000 7f9daae70640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:58:23.185 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:23.179+0000 7f9daae70640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:58:23.259 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:23.251+0000 7f9daae70640 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-02-05T06:58:29.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-02-05T06:58:29.748 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:58:29.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T06:58:29.750 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T06:58:29.750 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:58:30.134 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T06:58:30.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T06:58:30.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-02-05T06:58:30.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:58:30.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:58:30.135 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:58:30.164 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:30.155+0000 7f865adc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:30.166 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:30.159+0000 7f865adc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:30.167 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:30.159+0000 7f865adc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:58:30.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T06:58:30.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:58:30.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:58:30.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:58:30.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:58:30.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:58:30.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:58:30.469 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:58:30.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:58:30.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:58:30.739 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:30.731+0000 7f865adc8640 -1 Falling back to public interface 2024-02-05T06:58:30.802 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:58:31.803 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:58:31.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:58:31.804 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:58:31.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:58:31.804 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:58:31.805 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:58:32.112 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:58:33.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:58:33.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:58:33.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:58:33.114 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:58:33.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:58:33.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:58:33.277 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:58:33.271+0000 7f865adc8640 -1 osd.2 0 log_to_monitors true 2024-02-05T06:58:33.419 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:58:34.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:58:34.421 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:58:34.422 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:58:34.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:58:34.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:58:34.422 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:58:34.726 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/115578241,v1:127.0.0.1:6819/115578241] [v2:127.0.0.1:6820/115578241,v1:127.0.0.1:6821/115578241] exists,up 6e9a52a2-fe29-4b8c-8720-e7350ef12e61 2024-02-05T06:58:34.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:58:34.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:58:34.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:58:34.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:553: TEST_dump_scrub_schedule: create_pool test 1 1 2024-02-05T06:58:34.727 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-05T06:58:35.156 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'test' created 2024-02-05T06:58:35.176 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:58:36.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: wait_for_clean 2024-02-05T06:58:36.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:58:36.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:58:36.178 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:58:36.179 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:58:36.179 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:58:36.180 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:58:36.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:58:36.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:58:36.180 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:58:36.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:58:36.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:58:36.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:58:36.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:58:36.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=500 2024-02-05T06:58:36.302 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:58:36.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:58:36.754 INFO:tasks.workunit.client.0.smithi148.stderr:1 2024-02-05T06:58:36.754 INFO:tasks.workunit.client.0.smithi148.stderr:2' 2024-02-05T06:58:36.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:58:36.754 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:58:36.755 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:58:36.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-02-05T06:58:36.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-02-05T06:58:36.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-02-05T06:58:36.896 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:58:36.896 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:58:37.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-02-05T06:58:37.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-02-05T06:58:37.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2024-02-05T06:58:37.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:58:37.044 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:58:37.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-02-05T06:58:37.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-02-05T06:58:37.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2024-02-05T06:58:37.184 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:58:37.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-02-05T06:58:37.185 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:58:37.186 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:58:37.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-02-05T06:58:37.186 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:58:37.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-02-05T06:58:37.188 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836487 2024-02-05T06:58:37.188 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-02-05T06:58:37.188 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:58:37.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836487 2024-02-05T06:58:37.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:58:38.496 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 500 -eq 0 ']' 2024-02-05T06:58:38.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:58:38.814 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836487 2024-02-05T06:58:38.815 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:58:38.815 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-02-05T06:58:38.816 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:58:38.817 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:58:38.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:58:38.818 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-02-05T06:58:38.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-02-05T06:58:38.819 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-02-05T06:58:38.819 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672964 2024-02-05T06:58:38.820 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:58:39.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-02-05T06:58:39.137 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:58:39.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:58:39.138 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-02-05T06:58:39.139 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:58:39.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-02-05T06:58:39.139 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:58:39.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-02-05T06:58:39.141 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509442 2024-02-05T06:58:39.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-02-05T06:58:39.141 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:58:39.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-02-05T06:58:39.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:58:39.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:58:39.479 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:58:39.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:58:39.850 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:58:39.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:58:39.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:58:39.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:58:39.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:58:39.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:58:39.851 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:58:40.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:58:40.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:58:40.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:58:40.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:58:40.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:58:40.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:58:40.520 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:58:40.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: ceph osd dump 2024-02-05T06:58:40.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: grep '^pool.*['\'']test['\'']' 2024-02-05T06:58:40.521 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: awk '{ print $2 }' 2024-02-05T06:58:40.864 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: poolid=1 2024-02-05T06:58:40.865 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: dd if=/dev/urandom of=testdata.545011 bs=1032 count=1 2024-02-05T06:58:40.865 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records in 2024-02-05T06:58:40.865 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records out 2024-02-05T06:58:40.865 INFO:tasks.workunit.client.0.smithi148.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 8.9496e-05 s, 11.5 MB/s 2024-02-05T06:58:40.865 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: seq 1 15 2024-02-05T06:58:40.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:40.866 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj1 testdata.545011 2024-02-05T06:58:41.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:41.039 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj2 testdata.545011 2024-02-05T06:58:41.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:41.203 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj3 testdata.545011 2024-02-05T06:58:41.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:41.346 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj4 testdata.545011 2024-02-05T06:58:41.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:41.489 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj5 testdata.545011 2024-02-05T06:58:41.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:41.647 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj6 testdata.545011 2024-02-05T06:58:41.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:41.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj7 testdata.545011 2024-02-05T06:58:41.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:41.982 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj8 testdata.545011 2024-02-05T06:58:42.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:42.141 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj9 testdata.545011 2024-02-05T06:58:42.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:42.301 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj10 testdata.545011 2024-02-05T06:58:42.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:42.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj11 testdata.545011 2024-02-05T06:58:42.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:42.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj12 testdata.545011 2024-02-05T06:58:42.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:42.762 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj13 testdata.545011 2024-02-05T06:58:42.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:42.946 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj14 testdata.545011 2024-02-05T06:58:43.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:58:43.105 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj15 testdata.545011 2024-02-05T06:58:43.264 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rm -f testdata.545011 2024-02-05T06:58:43.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:564: TEST_dump_scrub_schedule: local pgid=1.0 2024-02-05T06:58:43.265 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: date -Ins 2024-02-05T06:58:43.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: local now_is=2024-02-05T06:58:43,264473158+00:00 2024-02-05T06:58:43.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:573: TEST_dump_scrub_schedule: expct_starting=(['query_active']='false' ['query_is_future']='true' ['query_schedule']='scrub scheduled') 2024-02-05T06:58:43.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:573: TEST_dump_scrub_schedule: declare -A expct_starting 2024-02-05T06:58:43.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:574: TEST_dump_scrub_schedule: declare -A sched_data 2024-02-05T06:58:43.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:575: TEST_dump_scrub_schedule: extract_published_sch 1.0 2024-02-05T06:58:43,264473158+00:00 2019-10-12T20:32:43.645168+0000 sched_data 2024-02-05T06:58:43.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.0 2024-02-05T06:58:43.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2024-02-05T06:58:43.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-02-05T06:58:43,264473158+00:00 2024-02-05T06:58:43.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2019-10-12T20:32:43.645168+0000 2024-02-05T06:58:43.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=1 2024-02-05T06:58:43.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-02-05T06:58:43.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-02-05T06:58:43.560 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:58:43.757 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T06:58:43.757 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=19 2024-02-05T06:58:43.757 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=29 2024-02-05T06:58:43.757 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T06:58:43.757 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:58:43.757 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T06:58:35.093' 2024-02-05T06:58:43.757 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=0 2024-02-05T06:58:43.757 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T06:58:35.093347+0000' 2024-02-05T06:58:43.757 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='0x0' 2024-02-05T06:58:43.757 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T06:58:43.757 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T06:58:43.758 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=null 2024-02-05T06:58:43.758 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:58:43.758 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:58:43.758 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=0 2024-02-05T06:58:43.758 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:58:43.758 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T06:58:35.093347+0000' 2024-02-05T06:58:43.758 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T06:58:43.758 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T06:58:43.758 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=18 2024-02-05T06:58:43.758 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=10 2024-02-05T06:58:43.758 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T06:58:43.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:576: TEST_dump_scrub_schedule: schedule_against_expected sched_data expct_starting initial 2024-02-05T06:58:43.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:170: schedule_against_expected: local -n dict=sched_data 2024-02-05T06:58:43.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:171: schedule_against_expected: local -n ep=expct_starting 2024-02-05T06:58:43.787 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:172: schedule_against_expected: local extr_dbg=1 2024-02-05T06:58:43.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:175: schedule_against_expected: local saved_echo_flag=x 2024-02-05T06:58:43.788 INFO:tasks.workunit.client.0.smithi148.stdout:-- - comparing: 2024-02-05T06:58:43.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:176: schedule_against_expected: set +x 2024-02-05T06:58:43.789 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_schedule expected: scrub scheduled in actual: scrub scheduled 2024-02-05T06:58:43.789 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_is_future expected: true in actual: true 2024-02-05T06:58:43.789 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_active expected: false in actual: false 2024-02-05T06:58:43.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:195: schedule_against_expected: return 0 2024-02-05T06:58:43.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:577: TEST_dump_scrub_schedule: (( 0 == 0 )) 2024-02-05T06:58:43.790 INFO:tasks.workunit.client.0.smithi148.stdout:last-scrub --- 0x0 2024-02-05T06:58:43.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:578: TEST_dump_scrub_schedule: echo 'last-scrub --- ' 0x0 2024-02-05T06:58:43.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:584: TEST_dump_scrub_schedule: saved_last_stamp=2024-02-05T06:58:35.093347+0000 2024-02-05T06:58:43.790 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:585: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 0 2024-02-05T06:58:43.921 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T06:58:43.921 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_sleep = '' " 2024-02-05T06:58:43.921 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:58:43.935 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T06:58:43.935 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_sleep = '' " 2024-02-05T06:58:43.935 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:58:43.948 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2: { 2024-02-05T06:58:43.948 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_sleep = '' " 2024-02-05T06:58:43.948 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:58:43.962 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:586: TEST_dump_scrub_schedule: ceph tell 1.0 deep-scrub 2024-02-05T06:58:44.089 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:58:44.089 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": true, 2024-02-05T06:58:44.089 INFO:tasks.workunit.client.0.smithi148.stdout: "must": true, 2024-02-05T06:58:44.089 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "0.000000" 2024-02-05T06:58:44.089 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:58:44.104 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:591: TEST_dump_scrub_schedule: sleep 5 2024-02-05T06:58:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:592: TEST_dump_scrub_schedule: sched_data=() 2024-02-05T06:58:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:593: TEST_dump_scrub_schedule: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2024-02-05T06:58:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:593: TEST_dump_scrub_schedule: declare -A expct_qry_duration 2024-02-05T06:58:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:594: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-02-05T06:58:35.093347+0000 expct_qry_duration 'WaitingAfterScrub ' sched_data 2024-02-05T06:58:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-02-05T06:58:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-02-05T06:58:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-02-05T06:58:35.093347+0000 2024-02-05T06:58:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_qry_duration 2024-02-05T06:58:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-02-05T06:58:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-02-05T06:58:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-02-05T06:58:49.107 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-02-05T06:58:49.108 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-02-05T06:58:49.108 INFO:tasks.workunit.client.0.smithi148.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.0 dt:2024-02-05T06:58:35.093347+0000 (10 retries) 2024-02-05T06:58:49.908 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:58:50.100 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T06:58:50.100 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=19 2024-02-05T06:58:50.100 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=39 2024-02-05T06:58:50.100 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T06:58:50.100 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T06:58:44.525' 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=1 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T06:58:44.525122+0000' 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='19x15' 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=null 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=1 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T06:58:44.525122+0000' 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=19 2024-02-05T06:58:50.101 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=38 2024-02-05T06:58:50.102 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T06:58:50.131 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 1 ~ false / 39 / 38 / true / 2024-02-05T06:58:44.525122+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T06:58:50.131 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2024-02-05T06:58:50.131 INFO:tasks.workunit.client.0.smithi148.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-02-05T06:58:50.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-02-05T06:58:50.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:596: TEST_dump_scrub_schedule: sched_data=() 2024-02-05T06:58:50.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:597: TEST_dump_scrub_schedule: expct_dmp_duration=(['dmp_last_duration']='0' ['dmp_last_duration_neg']='not0') 2024-02-05T06:58:50.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:597: TEST_dump_scrub_schedule: declare -A expct_dmp_duration 2024-02-05T06:58:50.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:598: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-02-05T06:58:35.093347+0000 expct_dmp_duration 'WaitingAfterScrub_dmp ' sched_data 2024-02-05T06:58:50.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-02-05T06:58:50.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-02-05T06:58:50.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-02-05T06:58:35.093347+0000 2024-02-05T06:58:50.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_dmp_duration 2024-02-05T06:58:50.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-02-05T06:58:50.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-02-05T06:58:50.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-02-05T06:58:50.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-02-05T06:58:50.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-02-05T06:58:50.133 INFO:tasks.workunit.client.0.smithi148.stdout:waiting for any condition (WaitingAfterScrub_dmp ): pg:1.0 dt:2024-02-05T06:58:35.093347+0000 (10 retries) 2024-02-05T06:58:50.927 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:58:51.125 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T06:58:51.125 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=19 2024-02-05T06:58:51.125 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=39 2024-02-05T06:58:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T06:58:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:58:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T06:58:44.525' 2024-02-05T06:58:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=1 2024-02-05T06:58:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T06:58:44.525122+0000' 2024-02-05T06:58:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='19x15' 2024-02-05T06:58:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T06:58:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T06:58:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=null 2024-02-05T06:58:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:58:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:58:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=1 2024-02-05T06:58:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:58:51.126 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T06:58:44.525122+0000' 2024-02-05T06:58:51.127 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T06:58:51.127 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T06:58:51.127 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=19 2024-02-05T06:58:51.127 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=38 2024-02-05T06:58:51.127 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T06:58:51.156 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 1 ~ false / 39 / 38 / true / 2024-02-05T06:58:44.525122+0000 / scrub scheduled %%% dmp_last_duration dmp_last_duration_neg 2024-02-05T06:58:51.156 INFO:tasks.workunit.client.0.smithi148.stdout:key is dmp_last_duration: negation:1 # expected: 0 # in actual: 1 2024-02-05T06:58:51.156 INFO:tasks.workunit.client.0.smithi148.stdout:WaitingAfterScrub_dmp - 'dmp_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-02-05T06:58:51.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-02-05T06:58:51.156 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:600: TEST_dump_scrub_schedule: sleep 2 2024-02-05T06:58:53.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:607: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_chunk_max 3 2024-02-05T06:58:53.298 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T06:58:53.298 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_chunk_max = '' (not observed, change may require restart) " 2024-02-05T06:58:53.298 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:58:53.311 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T06:58:53.311 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_chunk_max = '' (not observed, change may require restart) " 2024-02-05T06:58:53.312 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:58:53.325 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2: { 2024-02-05T06:58:53.325 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_chunk_max = '' (not observed, change may require restart) " 2024-02-05T06:58:53.325 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:58:53.339 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:608: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 1.0 2024-02-05T06:58:53.469 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0: { 2024-02-05T06:58:53.469 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_sleep = '' " 2024-02-05T06:58:53.469 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:58:53.482 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1: { 2024-02-05T06:58:53.482 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_sleep = '' " 2024-02-05T06:58:53.482 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:58:53.495 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2: { 2024-02-05T06:58:53.495 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_sleep = '' " 2024-02-05T06:58:53.495 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:58:53.510 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:609: TEST_dump_scrub_schedule: ceph osd set noscrub 2024-02-05T06:58:54.020 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T06:58:54.041 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:610: TEST_dump_scrub_schedule: sleep 2 2024-02-05T06:58:56.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:611: TEST_dump_scrub_schedule: saved_last_stamp=2024-02-05T06:58:44.525122+0000 2024-02-05T06:58:56.043 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:613: TEST_dump_scrub_schedule: ceph tell 1.0 schedule-scrub 2024-02-05T06:58:56.181 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:58:56.181 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": false, 2024-02-05T06:58:56.181 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T06:58:56.181 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-29T06:57:16.178560+0000" 2024-02-05T06:58:56.181 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:58:56.194 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:614: TEST_dump_scrub_schedule: sleep 1 2024-02-05T06:58:57.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:615: TEST_dump_scrub_schedule: sched_data=() 2024-02-05T06:58:57.197 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:616: TEST_dump_scrub_schedule: expct_scrub_peri_sched=(['query_is_future']='false') 2024-02-05T06:58:57.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:616: TEST_dump_scrub_schedule: declare -A expct_scrub_peri_sched 2024-02-05T06:58:57.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:617: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-02-05T06:58:44.525122+0000 expct_scrub_peri_sched waitingBeingScheduled sched_data 2024-02-05T06:58:57.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-02-05T06:58:57.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-02-05T06:58:57.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-02-05T06:58:44.525122+0000 2024-02-05T06:58:57.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_scrub_peri_sched 2024-02-05T06:58:57.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-02-05T06:58:57.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-02-05T06:58:57.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-02-05T06:58:57.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-02-05T06:58:57.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-02-05T06:58:57.199 INFO:tasks.workunit.client.0.smithi148.stdout:waiting for any condition (waitingBeingScheduled): pg:1.0 dt:2024-02-05T06:58:44.525122+0000 (10 retries) 2024-02-05T06:58:57.993 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:58:58.171 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T06:58:58.171 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=20 2024-02-05T06:58:58.171 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=44 2024-02-05T06:58:58.171 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T06:58:58.171 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='queued for scrub' 2024-02-05T06:58:58.171 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='0' 2024-02-05T06:58:58.171 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=1 2024-02-05T06:58:58.171 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-01-29T06:57:16.178560+0000' 2024-02-05T06:58:58.171 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='19x15' 2024-02-05T06:58:58.171 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=false 2024-02-05T06:58:58.171 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=false 2024-02-05T06:58:58.171 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=null 2024-02-05T06:58:58.171 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:58:58.172 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:58:58.172 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=1 2024-02-05T06:58:58.172 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:58:58.172 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T06:58:44.525122+0000' 2024-02-05T06:58:58.172 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T06:58:58.172 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T06:58:58.172 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=19 2024-02-05T06:58:58.172 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=39 2024-02-05T06:58:58.172 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T06:58:58.199 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 1 ~ false / 44 / 39 / false / 2024-01-29T06:57:16.178560+0000 / queued for scrub %%% query_is_future 2024-02-05T06:58:58.199 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_is_future: negation:0 # expected: false # in actual: false 2024-02-05T06:58:58.199 INFO:tasks.workunit.client.0.smithi148.stdout:waitingBeingScheduled - 'query_is_future' actual value (false) matches expected (false) (negation: 0) 2024-02-05T06:58:58.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-02-05T06:58:58.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:629: TEST_dump_scrub_schedule: saved_last_stamp=2024-01-29T06:57:16.178560+0000 2024-02-05T06:58:58.199 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:630: TEST_dump_scrub_schedule: ceph osd unset noscrub 2024-02-05T06:58:58.658 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is unset 2024-02-05T06:58:58.680 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:632: TEST_dump_scrub_schedule: cond_active=(['query_active']='true') 2024-02-05T06:58:58.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:632: TEST_dump_scrub_schedule: declare -A cond_active 2024-02-05T06:58:58.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:633: TEST_dump_scrub_schedule: sched_data=() 2024-02-05T06:58:58.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:634: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-01-29T06:57:16.178560+0000 cond_active 'WaitingActive ' sched_data 2024-02-05T06:58:58.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-02-05T06:58:58.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-02-05T06:58:58.681 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-01-29T06:57:16.178560+0000 2024-02-05T06:58:58.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=cond_active 2024-02-05T06:58:58.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-02-05T06:58:58.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-02-05T06:58:58.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-02-05T06:58:58.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-02-05T06:58:58.682 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-02-05T06:58:58.683 INFO:tasks.workunit.client.0.smithi148.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2024-01-29T06:57:16.178560+0000 (10 retries) 2024-02-05T06:58:59.474 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:58:59.667 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T06:58:59.667 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=21 2024-02-05T06:58:59.667 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=48 2024-02-05T06:58:59.667 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=true 2024-02-05T06:58:59.667 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrubbing' 2024-02-05T06:58:59.667 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='0' 2024-02-05T06:58:59.668 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=1 2024-02-05T06:58:59.668 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-01-29T06:57:16.178560+0000' 2024-02-05T06:58:59.668 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='19x15' 2024-02-05T06:58:59.668 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=false 2024-02-05T06:58:59.668 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=false 2024-02-05T06:58:59.668 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=null 2024-02-05T06:58:59.668 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:58:59.668 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:58:59.668 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=1 2024-02-05T06:58:59.668 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='queued for scrub' 2024-02-05T06:58:59.668 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='0' 2024-02-05T06:58:59.668 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=false 2024-02-05T06:58:59.669 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=false 2024-02-05T06:58:59.669 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=20 2024-02-05T06:58:59.669 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=43 2024-02-05T06:58:59.669 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T06:58:59.698 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 1 ~ true / 48 / 43 / false / 2024-01-29T06:57:16.178560+0000 / scrubbing %%% query_active 2024-02-05T06:58:59.698 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_active: negation:0 # expected: true # in actual: true 2024-02-05T06:58:59.698 INFO:tasks.workunit.client.0.smithi148.stdout:WaitingActive - 'query_active' actual value (true) matches expected (true) (negation: 0) 2024-02-05T06:58:59.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-02-05T06:58:59.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:638: TEST_dump_scrub_schedule: cond_active_dmp=(['dmp_state_has_scrubbing']='true' ['query_active']='false') 2024-02-05T06:58:59.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:638: TEST_dump_scrub_schedule: declare -A cond_active_dmp 2024-02-05T06:58:59.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:639: TEST_dump_scrub_schedule: sched_data=() 2024-02-05T06:58:59.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:640: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-01-29T06:57:16.178560+0000 cond_active_dmp 'WaitingActive ' sched_data 2024-02-05T06:58:59.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-02-05T06:58:59.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-02-05T06:58:59.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-01-29T06:57:16.178560+0000 2024-02-05T06:58:59.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=cond_active_dmp 2024-02-05T06:58:59.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-02-05T06:58:59.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-02-05T06:58:59.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-02-05T06:58:59.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-02-05T06:58:59.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-02-05T06:58:59.701 INFO:tasks.workunit.client.0.smithi148.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2024-01-29T06:57:16.178560+0000 (10 retries) 2024-02-05T06:59:00.513 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T06:59:00.701 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T06:59:00.701 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=21 2024-02-05T06:59:00.701 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=53 2024-02-05T06:59:00.701 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T06:59:00.701 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:59:00.701 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T06:59:00.595' 2024-02-05T06:59:00.701 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=2 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T06:59:00.595562+0000' 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='19x15' 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=null 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=1 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='queued for scrub' 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='0' 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=false 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=false 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=20 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=43 2024-02-05T06:59:00.702 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T06:59:00.730 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 1 ~ false / 53 / 43 / true / 2024-02-05T06:59:00.595562+0000 / scrub scheduled %%% dmp_state_has_scrubbing query_active 2024-02-05T06:59:00.730 INFO:tasks.workunit.client.0.smithi148.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-02-05T06:59:00.730 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_active: negation:0 # expected: false # in actual: false 2024-02-05T06:59:00.730 INFO:tasks.workunit.client.0.smithi148.stdout:WaitingActive - 'query_active' actual value (false) matches expected (false) (negation: 0) 2024-02-05T06:59:00.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-02-05T06:59:00.730 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_dump_scrub_schedule: perf_counters td/osd-scrub-test 3 2024-02-05T06:59:00.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-02-05T06:59:00.731 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-02-05T06:59:00.731 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-02-05T06:59:00.731 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-02-05T06:59:00.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:59:00.732 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-02-05T06:59:00.733 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T06:59:00.865 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:59:00.865 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_ec": [ 2024-02-05T06:59:00.865 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:59:00.865 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:59:00.865 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T06:59:00.865 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T06:59:00.865 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.865 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:59:00.865 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:59:00.865 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:59:00.865 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:59:00.865 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:59:00.865 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:59:00.866 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:00.867 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:00.868 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:59:00.868 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_repl": [ 2024-02-05T06:59:00.868 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:59:00.868 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:59:00.868 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T06:59:00.868 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T06:59:00.868 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.868 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:59:00.868 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:59:00.868 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:59:00.868 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:59:00.868 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:59:00.868 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:59:00.869 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:00.870 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_ec": [ 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.871 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.872 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.872 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:59:00.872 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:59:00.872 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.872 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.872 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.872 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.872 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:59:00.872 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:59:00.872 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:59:00.872 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:59:00.872 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:59:00.872 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:59:00.873 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_repl": [ 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.874 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.875 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:59:00.875 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:59:00.875 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.875 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.875 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.875 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.875 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:59:00.875 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:59:00.875 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:59:00.875 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:59:00.878 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:59:00.878 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:59:00.878 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:59:00.878 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.878 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.878 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.878 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.878 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:59:00.878 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:59:00.878 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:59:00.878 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:59:00.878 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:59:00.878 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:00.878 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:00.879 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:00.879 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:00.879 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:59:00.879 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:00.879 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:00.879 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:59:00.879 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:59:00.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:59:00.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T06:59:00.879 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-02-05T06:59:01.005 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:59:01.005 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_ec": [ 2024-02-05T06:59:01.005 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:59:01.005 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:59:01.005 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T06:59:01.005 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T06:59:01.005 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.005 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:59:01.005 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:59:01.005 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:59:01.005 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:59:01.006 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.007 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.008 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.008 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:59:01.008 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.008 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.008 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:59:01.008 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_repl": [ 2024-02-05T06:59:01.008 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:59:01.008 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:59:01.008 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T06:59:01.008 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T06:59:01.008 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.008 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:59:01.008 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 1, 2024-02-05T06:59:01.008 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 1, 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 1, 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 1, 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0.039999027, 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0.039999027 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 1, 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:59:01.009 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 1, 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.010 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 2 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_ec": [ 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:59:01.011 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:59:01.012 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.012 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.012 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.012 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.012 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:59:01.012 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:59:01.012 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.012 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.012 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.012 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.012 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:59:01.012 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:59:01.012 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.013 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.014 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.014 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.014 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:59:01.014 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.014 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.014 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:59:01.014 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_repl": [ 2024-02-05T06:59:01.014 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:59:01.014 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:59:01.014 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T06:59:01.014 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T06:59:01.014 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.014 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:59:01.014 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 1, 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 1, 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 1, 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 1, 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 1.071973956, 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 1.071973956 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 1, 2024-02-05T06:59:01.015 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:59:01.017 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:59:01.017 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:59:01.017 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 1, 2024-02-05T06:59:01.017 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:59:01.017 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 1, 2024-02-05T06:59:01.017 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0.067998349, 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0.067998349 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 2 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.018 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:59:01.019 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:59:01.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:59:01.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-02-05T06:59:01.019 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T06:59:01.143 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T06:59:01.143 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_ec": [ 2024-02-05T06:59:01.143 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:59:01.143 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:59:01.143 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T06:59:01.143 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T06:59:01.143 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.143 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:59:01.143 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:59:01.144 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:59:01.144 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:59:01.144 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:59:01.144 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.144 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.144 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.144 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.144 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:59:01.144 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:59:01.144 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.144 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.144 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.144 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:59:01.145 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:59:01.146 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:59:01.146 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.146 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.146 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.146 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.146 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:59:01.146 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.146 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.146 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:59:01.146 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_repl": [ 2024-02-05T06:59:01.146 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:59:01.146 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:59:01.146 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T06:59:01.146 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.147 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.148 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.148 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:59:01.148 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:59:01.148 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:59:01.148 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:59:01.148 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:59:01.148 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:59:01.148 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:59:01.148 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.148 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.148 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.148 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.148 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:59:01.148 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_ec": [ 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:59:01.149 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.150 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:59:01.151 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:59:01.152 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:59:01.152 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:59:01.152 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:59:01.152 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.152 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.152 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.152 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.152 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:59:01.152 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.152 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.152 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T06:59:01.152 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_repl": [ 2024-02-05T06:59:01.152 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T06:59:01.152 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:59:01.153 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.154 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.154 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.154 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.154 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T06:59:01.154 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T06:59:01.154 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T06:59:01.154 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T06:59:01.156 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:59:01.156 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:59:01.156 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T06:59:01.156 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.156 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.156 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.156 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.156 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T06:59:01.156 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:59:01.156 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T06:59:01.156 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T06:59:01.156 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T06:59:01.157 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T06:59:01.157 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T06:59:01.157 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T06:59:01.157 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T06:59:01.157 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T06:59:01.157 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.157 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T06:59:01.157 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T06:59:01.157 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T06:59:01.157 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_dump_scrub_schedule ------------------' 2024-02-05T06:59:01.158 INFO:tasks.workunit.client.0.smithi148.stdout:-------------- Teardown Test TEST_dump_scrub_schedule ------------------ 2024-02-05T06:59:01.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-02-05T06:59:01.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T06:59:01.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:59:01.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T06:59:01.159 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:59:01.159 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:59:01.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:59:01.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:59:01.159 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:59:01.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:59:01.467 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:59:01.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:59:01.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:59:01.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:59:01.469 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:59:01.470 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:59:01.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:59:01.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:59:01.471 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:59:01.471 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:59:01.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:59:01.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:59:01.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T06:59:01.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:59:01.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:59:01.482 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:59:01.482 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.545011 2024-02-05T06:59:01.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:59:01.483 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:59:01.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_dump_scrub_schedule ------------------' 2024-02-05T06:59:01.484 INFO:tasks.workunit.client.0.smithi148.stdout:-------------- Complete Test TEST_dump_scrub_schedule ------------------ 2024-02-05T06:59:01.484 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-02-05T06:59:01.484 INFO:tasks.workunit.client.0.smithi148.stdout:-------------- Prepare Test TEST_interval_changes ------------------- 2024-02-05T06:59:01.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_interval_changes -------------------' 2024-02-05T06:59:01.485 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-02-05T06:59:01.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-02-05T06:59:01.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-02-05T06:59:01.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T06:59:01.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:59:01.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T06:59:01.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:59:01.486 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:59:01.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:59:01.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:59:01.486 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:59:01.487 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:59:01.487 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:59:01.488 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:59:01.489 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:59:01.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:59:01.490 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:59:01.491 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:59:01.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:59:01.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:59:01.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:59:01.492 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:59:01.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:59:01.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:59:01.495 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T06:59:01.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:59:01.496 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:59:01.497 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:59:01.497 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.545011 2024-02-05T06:59:01.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:59:01.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:59:01.498 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-02-05T06:59:01.499 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:59:01.499 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:59:01.500 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:59:01.500 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.545011 2024-02-05T06:59:01.501 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:59:01.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:59:01.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:59:01.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-02-05T06:59:01.502 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_interval_changes -----------------------' 2024-02-05T06:59:01.502 INFO:tasks.workunit.client.0.smithi148.stdout:-------------- Run Test TEST_interval_changes ----------------------- 2024-02-05T06:59:01.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_interval_changes td/osd-scrub-test 2024-02-05T06:59:01.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:149: TEST_interval_changes: local poolname=test 2024-02-05T06:59:01.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:150: TEST_interval_changes: local OSDS=2 2024-02-05T06:59:01.503 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:151: TEST_interval_changes: local objects=10 2024-02-05T06:59:01.503 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:153: TEST_interval_changes: expr 24 '*' 60 '*' 60 2024-02-05T06:59:01.504 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:153: TEST_interval_changes: local day=86400 2024-02-05T06:59:01.505 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:154: TEST_interval_changes: expr 86400 '*' 7 2024-02-05T06:59:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:154: TEST_interval_changes: local week=604800 2024-02-05T06:59:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: local min_interval=86400 2024-02-05T06:59:01.506 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:156: TEST_interval_changes: local max_interval=604800 2024-02-05T06:59:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: local WAIT_FOR_UPDATE=15 2024-02-05T06:59:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:159: TEST_interval_changes: TESTDATA=testdata.545011 2024-02-05T06:59:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:162: TEST_interval_changes: run_mon td/osd-scrub-test a --osd_pool_default_size=2 2024-02-05T06:59:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-02-05T06:59:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:59:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:59:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:59:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-02-05T06:59:01.507 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=2 2024-02-05T06:59:01.897 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:59:01.897 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:59:01.897 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:59:01.898 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:59:01.898 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:59:01.898 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:59:01.898 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:59:01.899 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T06:59:02.143 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:59:02.144 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:59:02.145 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:59:02.145 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:59:02.145 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:59:02.146 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:59:02.146 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:59:02.146 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:59:02.146 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:59:02.146 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:59:02.146 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:59:02.146 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:59:02.147 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.545011/ceph-mon.a.asok 2024-02-05T06:59:02.147 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:59:02.147 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.545011/ceph-mon.a.asok config get fsid 2024-02-05T06:59:02.263 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:59:02.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:59:02.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:59:02.264 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:59:02.265 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:59:02.265 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:59:02.265 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:59:02.265 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:59:02.265 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:59:02.266 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:59:02.266 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:59:02.266 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.545011/ceph-mon.a.asok 2024-02-05T06:59:02.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:59:02.266 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.545011/ceph-mon.a.asok config get mon_host 2024-02-05T06:59:02.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:163: TEST_interval_changes: run_mgr td/osd-scrub-test x 2024-02-05T06:59:02.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-02-05T06:59:02.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:59:02.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:59:02.370 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:59:02.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-02-05T06:59:02.371 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:59:03.225 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:59:03.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:59:03.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:59:03.226 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:59:03.226 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:59:03.226 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:59:03.226 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:59:03.228 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:59:03.229 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:59:03.262 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: expr 2 - 1 2024-02-05T06:59:03.262 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: seq 0 1 2024-02-05T06:59:03.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:59:03.266 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:166: TEST_interval_changes: run_osd td/osd-scrub-test 0 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:59:03.267 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:59:03.268 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:59:03.268 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:59:03.268 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:59:03.268 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:59:03.268 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:59:03.268 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:59:03.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:59:03.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:59:03.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:59:03.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:59:03.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:59:03.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:59:03.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:59:03.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:59:03.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:59:03.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:59:03.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:59:03.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2024-02-05T06:59:03.269 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-02-05T06:59:03.269 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:59:03.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1eda8c6f-85ba-4da9-bb6d-581d4fa11fdc 2024-02-05T06:59:03.270 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1eda8c6f-85ba-4da9-bb6d-581d4fa11fdc' 2024-02-05T06:59:03.270 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 1eda8c6f-85ba-4da9-bb6d-581d4fa11fdc 2024-02-05T06:59:03.271 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:59:03.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3h8BlyMW+EBAAOU613GVCAah835UCVoRp/g== 2024-02-05T06:59:03.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3h8BlyMW+EBAAOU613GVCAah835UCVoRp/g=="}' 2024-02-05T06:59:03.284 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1eda8c6f-85ba-4da9-bb6d-581d4fa11fdc -i td/osd-scrub-test/0/new.json 2024-02-05T06:59:03.546 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:59:03.562 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-02-05T06:59:03.563 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQA3h8BlyMW+EBAAOU613GVCAah835UCVoRp/g== --osd-uuid 1eda8c6f-85ba-4da9-bb6d-581d4fa11fdc 2024-02-05T06:59:03.598 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:03.590+0000 7ff201e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:59:03.600 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:03.594+0000 7ff201e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:59:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:03.594+0000 7ff201e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:59:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:03.594+0000 7ff201e13640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:59:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:03.594+0000 7ff201e13640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:59:03.602 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:03.594+0000 7ff201e13640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:59:03.690 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:03.682+0000 7ff201e13640 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-02-05T06:59:09.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-02-05T06:59:09.650 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:59:09.651 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T06:59:09.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:59:09.651 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:59:10.034 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T06:59:10.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:59:10.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-02-05T06:59:10.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:59:10.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:59:10.035 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:59:10.067 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:10.058+0000 7f529b4d0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:59:10.096 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:10.086+0000 7f529b4d0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:59:10.097 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:10.090+0000 7f529b4d0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:59:10.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:59:10.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:59:10.354 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:59:10.355 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:59:10.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:59:10.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:59:10.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:10.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:59:10.355 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:10.356 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:10.643 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:10.634+0000 7f529b4d0640 -1 Falling back to public interface 2024-02-05T06:59:10.673 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:11.674 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:11.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:11.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:59:11.675 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:59:11.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:11.675 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:11.972 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:12.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:12.974 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:12.975 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:59:12.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:59:12.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:12.975 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:13.184 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:13.178+0000 7f529b4d0640 -1 osd.0 0 log_to_monitors true 2024-02-05T06:59:13.288 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:14.289 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:14.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:14.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:59:14.290 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:59:14.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:14.290 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:14.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:15.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:15.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:15.606 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:59:15.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:59:15.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:15.607 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:15.925 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:16.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:16.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:16.926 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:59:16.927 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:59:16.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:16.927 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:17.273 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:18.274 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:18.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:18.275 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:59:18.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:59:18.275 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:18.276 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:18.580 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:19.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:19.581 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:19.582 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:59:19.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:59:19.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:19.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:19.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:20.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:20.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:20.989 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:59:20.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:59:20.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:20.990 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:21.305 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:22.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:22.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:22.308 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:59:22.308 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:59:22.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:22.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:22.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:23.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:23.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:23.631 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:59:23.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:59:23.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:23.632 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:23.985 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:24.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:24.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:24.989 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T06:59:24.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:59:24.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:24.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:25.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:26.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:26.302 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:26.303 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T06:59:26.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:59:26.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:26.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:26.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:27.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:27.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:27.938 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T06:59:27.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:59:27.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:27.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:28.662 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:29.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:29.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:29.665 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T06:59:29.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:59:29.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:29.665 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:30.492 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:31.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:31.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:31.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:59:31.494 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T06:59:31.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:31.494 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:32.340 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:33.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:33.342 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:33.343 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T06:59:33.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:59:33.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:33.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:33.859 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:34.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:34.861 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:34.862 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T06:59:34.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:59:34.862 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:34.863 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:35.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:36.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:36.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:36.310 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T06:59:36.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:59:36.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:36.311 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:36.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:37.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:37.620 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:37.620 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T06:59:37.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:59:37.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:37.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:59:37.929 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1759108830,v1:127.0.0.1:6803/1759108830] [v2:127.0.0.1:6804/1759108830,v1:127.0.0.1:6805/1759108830] exists,up 1eda8c6f-85ba-4da9-bb6d-581d4fa11fdc 2024-02-05T06:59:37.934 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:59:37.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:59:37.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:59:37.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:59:37.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:166: TEST_interval_changes: run_osd td/osd-scrub-test 1 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-02-05T06:59:37.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:59:37.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:59:37.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:59:37.935 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:59:37.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-02-05T06:59:37.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:59:37.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:59:37.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:59:37.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:59:37.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-02-05T06:59:37.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-02-05T06:59:37.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:59:37.936 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:59:37.937 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:59:37.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:59:37.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:59:37.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:59:37.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2024-02-05T06:59:37.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-02-05T06:59:37.938 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:59:37.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3fdb226f-e691-48b1-8e62-a908581bc4ee 2024-02-05T06:59:37.938 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3fdb226f-e691-48b1-8e62-a908581bc4ee' 2024-02-05T06:59:37.938 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 3fdb226f-e691-48b1-8e62-a908581bc4ee 2024-02-05T06:59:37.938 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:59:37.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZh8BlZfmGOBAA9keVEIZlgAjUvVduoDbS2A== 2024-02-05T06:59:37.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZh8BlZfmGOBAA9keVEIZlgAjUvVduoDbS2A=="}' 2024-02-05T06:59:37.952 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3fdb226f-e691-48b1-8e62-a908581bc4ee -i td/osd-scrub-test/1/new.json 2024-02-05T06:59:38.310 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:59:38.329 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-02-05T06:59:38.331 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQBZh8BlZfmGOBAA9keVEIZlgAjUvVduoDbS2A== --osd-uuid 3fdb226f-e691-48b1-8e62-a908581bc4ee 2024-02-05T06:59:38.366 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:38.361+0000 7f81fb53d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:59:38.403 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:38.397+0000 7f81fb53d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:59:38.405 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:38.401+0000 7f81fb53d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:59:38.406 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:38.401+0000 7f81fb53d640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:59:38.406 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:38.401+0000 7f81fb53d640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:59:38.406 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:38.401+0000 7f81fb53d640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:59:38.467 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:38.461+0000 7f81fb53d640 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-02-05T06:59:45.006 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-02-05T06:59:45.007 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:59:45.008 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T06:59:45.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:59:45.009 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:59:45.379 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:59:45.379 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T06:59:45.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=7257fb16-9a94-4aea-ae2b-45cca029d367 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.545011/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-02-05T06:59:45.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:59:45.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:59:45.380 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:59:45.411 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:45.405+0000 7f5c0a401640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:59:45.413 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:45.409+0000 7f5c0a401640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:59:45.414 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:45.409+0000 7f5c0a401640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:59:45.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:59:45.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:59:45.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:59:45.699 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:59:45.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:59:45.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:45.700 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T06:59:45.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:59:45.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:45.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:59:45.966 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:45.961+0000 7f5c0a401640 -1 Falling back to public interface 2024-02-05T06:59:45.996 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:46.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:46.998 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:46.998 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T06:59:46.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:59:46.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:46.999 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:59:47.294 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:48.296 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:48.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:48.297 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T06:59:48.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:59:48.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:48.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:59:48.610 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:48.748 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T06:59:48.741+0000 7f5c0a401640 -1 osd.1 0 log_to_monitors true 2024-02-05T06:59:49.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:49.612 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:49.612 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T06:59:49.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:59:49.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:49.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:59:50.030 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:51.032 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:51.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:51.033 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T06:59:51.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:59:51.033 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:51.034 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:59:51.345 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:52.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:52.347 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:52.347 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T06:59:52.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:59:52.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:52.348 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:59:52.654 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:53.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:53.656 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:53.656 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T06:59:53.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:59:53.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:53.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:59:53.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:54.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:54.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:54.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:59:54.989 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T06:59:54.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:54.989 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:59:55.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:56.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:56.299 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:56.299 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T06:59:56.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:59:56.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:56.300 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:59:56.628 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:57.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:57.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:57.630 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T06:59:57.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:59:57.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:57.631 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:59:57.951 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:59:58.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:59:58.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:59:58.954 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T06:59:58.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:59:58.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:59:58.954 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:59:59.265 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T07:00:00.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T07:00:00.267 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T07:00:00.267 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T07:00:00.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T07:00:00.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T07:00:00.268 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T07:00:00.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T07:00:01.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T07:00:01.623 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T07:00:01.624 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T07:00:01.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T07:00:01.624 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T07:00:01.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T07:00:02.130 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T07:00:03.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T07:00:03.132 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T07:00:03.132 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T07:00:03.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T07:00:03.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T07:00:03.133 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T07:00:03.854 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T07:00:04.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T07:00:04.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T07:00:04.856 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T07:00:04.856 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T07:00:04.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T07:00:04.857 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T07:00:05.807 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T07:00:06.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T07:00:06.808 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T07:00:06.809 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T07:00:06.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T07:00:06.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T07:00:06.809 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T07:00:07.627 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T07:00:08.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T07:00:08.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T07:00:08.629 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T07:00:08.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T07:00:08.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T07:00:08.630 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T07:00:09.472 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T07:00:10.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T07:00:10.473 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T07:00:10.474 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T07:00:10.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T07:00:10.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T07:00:10.474 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T07:00:10.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T07:00:11.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T07:00:11.788 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T07:00:11.788 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T07:00:11.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T07:00:11.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T07:00:11.789 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T07:00:12.611 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T07:00:13.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T07:00:13.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T07:00:13.614 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T07:00:13.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T07:00:13.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T07:00:13.614 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T07:00:14.024 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2444025713,v1:127.0.0.1:6811/2444025713] [v2:127.0.0.1:6812/2444025713,v1:127.0.0.1:6813/2444025713] exists,up 3fdb226f-e691-48b1-8e62-a908581bc4ee 2024-02-05T07:00:14.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T07:00:14.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T07:00:14.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T07:00:14.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: create_pool test 1 1 2024-02-05T07:00:14.025 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-05T07:00:14.471 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'test' created 2024-02-05T07:00:14.491 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T07:00:15.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:171: TEST_interval_changes: wait_for_clean 2024-02-05T07:00:15.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T07:00:15.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T07:00:15.493 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T07:00:15.494 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T07:00:15.495 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T07:00:15.495 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T07:00:15.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T07:00:15.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T07:00:15.495 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T07:00:15.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T07:00:15.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T07:00:15.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T07:00:15.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T07:00:15.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=500 2024-02-05T07:00:15.618 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T07:00:16.000 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T07:00:16.000 INFO:tasks.workunit.client.0.smithi148.stderr:1' 2024-02-05T07:00:16.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T07:00:16.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T07:00:16.001 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T07:00:16.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2024-02-05T07:00:16.164 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2024-02-05T07:00:16.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2024-02-05T07:00:16.165 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T07:00:16.165 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T07:00:16.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-02-05T07:00:16.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-02-05T07:00:16.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672974' 2024-02-05T07:00:16.303 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T07:00:16.303 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2024-02-05T07:00:16.304 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T07:00:16.304 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T07:00:16.305 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2024-02-05T07:00:16.305 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T07:00:16.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2024-02-05T07:00:16.306 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-02-05T07:00:16.306 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836494 2024-02-05T07:00:16.306 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T07:00:16.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-02-05T07:00:16.605 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T07:00:17.606 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 500 -eq 0 ']' 2024-02-05T07:00:17.607 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T07:00:17.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836494 2024-02-05T07:00:17.904 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T07:00:17.904 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-02-05T07:00:17.904 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T07:00:17.906 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T07:00:17.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-02-05T07:00:17.907 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T07:00:17.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-02-05T07:00:17.908 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672974 2024-02-05T07:00:17.908 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-02-05T07:00:17.909 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T07:00:18.216 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-02-05T07:00:18.216 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T07:00:18.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T07:00:18.217 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T07:00:18.594 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T07:00:18.595 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T07:00:18.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T07:00:18.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T07:00:18.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T07:00:18.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T07:00:18.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T07:00:18.595 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T07:00:18.924 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T07:00:18.925 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T07:00:18.925 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T07:00:18.925 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T07:00:19.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T07:00:19.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T07:00:19.287 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T07:00:19.287 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: ceph osd dump 2024-02-05T07:00:19.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: grep '^pool.*['\'']test['\'']' 2024-02-05T07:00:19.288 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: awk '{ print $2 }' 2024-02-05T07:00:19.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: local poolid=1 2024-02-05T07:00:19.601 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:174: TEST_interval_changes: dd if=/dev/urandom of=testdata.545011 bs=1032 count=1 2024-02-05T07:00:19.602 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records in 2024-02-05T07:00:19.602 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records out 2024-02-05T07:00:19.603 INFO:tasks.workunit.client.0.smithi148.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000199996 s, 5.2 MB/s 2024-02-05T07:00:19.603 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: seq 1 10 2024-02-05T07:00:19.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T07:00:19.604 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj1 testdata.545011 2024-02-05T07:00:19.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T07:00:19.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj2 testdata.545011 2024-02-05T07:00:19.931 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T07:00:19.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj3 testdata.545011 2024-02-05T07:00:20.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T07:00:20.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj4 testdata.545011 2024-02-05T07:00:20.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T07:00:20.215 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj5 testdata.545011 2024-02-05T07:00:20.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T07:00:20.374 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj6 testdata.545011 2024-02-05T07:00:20.551 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T07:00:20.552 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj7 testdata.545011 2024-02-05T07:00:20.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T07:00:20.702 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj8 testdata.545011 2024-02-05T07:00:20.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T07:00:20.851 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj9 testdata.545011 2024-02-05T07:00:21.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T07:00:21.002 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj10 testdata.545011 2024-02-05T07:00:21.152 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:179: TEST_interval_changes: rm -f testdata.545011 2024-02-05T07:00:21.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:181: TEST_interval_changes: get_primary test obj1 2024-02-05T07:00:21.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-02-05T07:00:21.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T07:00:21.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T07:00:21.155 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-02-05T07:00:21.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:181: TEST_interval_changes: local primary=1 2024-02-05T07:00:21.466 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:184: TEST_interval_changes: check_dump_scrubs 1 '1 day' '1 week' 2024-02-05T07:00:21.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-02-05T07:00:21.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=1 day' 2024-02-05T07:00:21.467 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=1 week' 2024-02-05T07:00:21.468 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-02-05T07:00:21.468 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T07:00:21.468 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T07:00:21.468 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T07:00:21.468 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T07:00:21.468 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T07:00:21.468 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.545011/ceph-osd.1.asok 2024-02-05T07:00:21.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-02-05T07:00:21.468 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.545011/ceph-osd.1.asok dump_scrubs 2024-02-05T07:00:21.585 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-02-05T07:00:21.585 INFO:tasks.workunit.client.0.smithi148.stderr: { 2024-02-05T07:00:21.585 INFO:tasks.workunit.client.0.smithi148.stderr: "pgid": "1.0", 2024-02-05T07:00:21.586 INFO:tasks.workunit.client.0.smithi148.stderr: "sched_time": "2024-02-06T07:00:14.421369+0000", 2024-02-05T07:00:21.586 INFO:tasks.workunit.client.0.smithi148.stderr: "deadline": "2024-02-12T07:00:14.421369+0000", 2024-02-05T07:00:21.586 INFO:tasks.workunit.client.0.smithi148.stderr: "forced": false 2024-02-05T07:00:21.586 INFO:tasks.workunit.client.0.smithi148.stderr: } 2024-02-05T07:00:21.586 INFO:tasks.workunit.client.0.smithi148.stderr:]' 2024-02-05T07:00:21.586 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-06T07:00:14.421369+0000",' '"deadline":' '"2024-02-12T07:00:14.421369+0000",' '"forced":' false '}' ']' 2024-02-05T07:00:21.587 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-02-05T07:00:21.621 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-02-06T07:00:14.421369+0000"' 2024-02-05T07:00:21.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-02-06T07:00:14.421369+0000 2024-02-05T07:00:21.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-02-06T07:00:14.421369+0000 2024-02-05T07:00:21.622 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T07:00:21.624 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 day' 2024-02-05T07:00:21.625 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-02-06 = 2024-02-06 2024-02-05T07:00:21.626 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-06T07:00:14.421369+0000",' '"deadline":' '"2024-02-12T07:00:14.421369+0000",' '"forced":' false '}' ']' 2024-02-05T07:00:21.627 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-02-05T07:00:21.657 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-02-12T07:00:14.421369+0000"' 2024-02-05T07:00:21.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-02-12T07:00:14.421369+0000 2024-02-05T07:00:21.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T07:00:21.658 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-02-12T07:00:14.421369+0000 2024-02-05T07:00:21.660 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 week' 2024-02-05T07:00:21.660 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-02-12 = 2024-02-12 2024-02-05T07:00:21.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: get_asok_path osd.1 2024-02-05T07:00:21.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T07:00:21.661 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T07:00:21.662 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T07:00:21.662 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T07:00:21.662 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T07:00:21.662 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.545011/ceph-osd.1.asok 2024-02-05T07:00:21.663 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: expr 86400 '*' 2 2024-02-05T07:00:21.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: CEPH_ARGS= 2024-02-05T07:00:21.664 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.545011/ceph-osd.1.asok config set osd_scrub_min_interval 172800 2024-02-05T07:00:21.768 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T07:00:21.768 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_min_interval = '' " 2024-02-05T07:00:21.768 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T07:00:21.782 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:188: TEST_interval_changes: sleep 15 2024-02-05T07:00:36.784 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:189: TEST_interval_changes: check_dump_scrubs 1 '2 days' '1 week' 2024-02-05T07:00:36.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-02-05T07:00:36.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=2 days' 2024-02-05T07:00:36.785 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=1 week' 2024-02-05T07:00:36.786 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-02-05T07:00:36.786 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T07:00:36.786 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T07:00:36.787 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T07:00:36.787 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T07:00:36.787 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T07:00:36.787 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.545011/ceph-osd.1.asok 2024-02-05T07:00:36.788 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-02-05T07:00:36.788 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.545011/ceph-osd.1.asok dump_scrubs 2024-02-05T07:00:36.915 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-02-05T07:00:36.915 INFO:tasks.workunit.client.0.smithi148.stderr: { 2024-02-05T07:00:36.916 INFO:tasks.workunit.client.0.smithi148.stderr: "pgid": "1.0", 2024-02-05T07:00:36.916 INFO:tasks.workunit.client.0.smithi148.stderr: "sched_time": "2024-02-07T07:00:14.421369+0000", 2024-02-05T07:00:36.916 INFO:tasks.workunit.client.0.smithi148.stderr: "deadline": "2024-02-12T07:00:14.421369+0000", 2024-02-05T07:00:36.916 INFO:tasks.workunit.client.0.smithi148.stderr: "forced": false 2024-02-05T07:00:36.916 INFO:tasks.workunit.client.0.smithi148.stderr: } 2024-02-05T07:00:36.916 INFO:tasks.workunit.client.0.smithi148.stderr:]' 2024-02-05T07:00:36.916 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-07T07:00:14.421369+0000",' '"deadline":' '"2024-02-12T07:00:14.421369+0000",' '"forced":' false '}' ']' 2024-02-05T07:00:36.917 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-02-05T07:00:36.953 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-02-07T07:00:14.421369+0000"' 2024-02-05T07:00:36.953 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-02-07T07:00:14.421369+0000 2024-02-05T07:00:36.953 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T07:00:36.954 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-02-07T07:00:14.421369+0000 2024-02-05T07:00:36.955 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2024-02-05T07:00:36.957 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-02-07 = 2024-02-07 2024-02-05T07:00:36.958 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-07T07:00:14.421369+0000",' '"deadline":' '"2024-02-12T07:00:14.421369+0000",' '"forced":' false '}' ']' 2024-02-05T07:00:36.958 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-02-05T07:00:36.993 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-02-12T07:00:14.421369+0000"' 2024-02-05T07:00:36.993 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-02-12T07:00:14.421369+0000 2024-02-05T07:00:36.993 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-02-12T07:00:14.421369+0000 2024-02-05T07:00:36.994 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T07:00:36.997 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 week' 2024-02-05T07:00:36.997 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-02-12 = 2024-02-12 2024-02-05T07:00:36.998 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: get_asok_path osd.1 2024-02-05T07:00:36.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T07:00:36.999 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T07:00:36.999 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T07:00:37.000 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T07:00:37.000 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T07:00:37.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.545011/ceph-osd.1.asok 2024-02-05T07:00:37.000 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: expr 604800 '*' 2 2024-02-05T07:00:37.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: CEPH_ARGS= 2024-02-05T07:00:37.001 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.545011/ceph-osd.1.asok config set osd_scrub_max_interval 1209600 2024-02-05T07:00:37.110 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T07:00:37.111 INFO:tasks.workunit.client.0.smithi148.stdout: "success": "osd_scrub_max_interval = '' " 2024-02-05T07:00:37.111 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T07:00:37.123 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:193: TEST_interval_changes: sleep 15 2024-02-05T07:00:52.125 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:194: TEST_interval_changes: check_dump_scrubs 1 '2 days' '2 week' 2024-02-05T07:00:52.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-02-05T07:00:52.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=2 days' 2024-02-05T07:00:52.126 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=2 week' 2024-02-05T07:00:52.127 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-02-05T07:00:52.127 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T07:00:52.127 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T07:00:52.128 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T07:00:52.128 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T07:00:52.128 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T07:00:52.128 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.545011/ceph-osd.1.asok 2024-02-05T07:00:52.128 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-02-05T07:00:52.128 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.545011/ceph-osd.1.asok dump_scrubs 2024-02-05T07:00:52.256 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-02-05T07:00:52.257 INFO:tasks.workunit.client.0.smithi148.stderr: { 2024-02-05T07:00:52.257 INFO:tasks.workunit.client.0.smithi148.stderr: "pgid": "1.0", 2024-02-05T07:00:52.257 INFO:tasks.workunit.client.0.smithi148.stderr: "sched_time": "2024-02-07T07:00:14.421369+0000", 2024-02-05T07:00:52.257 INFO:tasks.workunit.client.0.smithi148.stderr: "deadline": "2024-02-19T07:00:14.421369+0000", 2024-02-05T07:00:52.257 INFO:tasks.workunit.client.0.smithi148.stderr: "forced": false 2024-02-05T07:00:52.257 INFO:tasks.workunit.client.0.smithi148.stderr: } 2024-02-05T07:00:52.257 INFO:tasks.workunit.client.0.smithi148.stderr:]' 2024-02-05T07:00:52.258 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-07T07:00:14.421369+0000",' '"deadline":' '"2024-02-19T07:00:14.421369+0000",' '"forced":' false '}' ']' 2024-02-05T07:00:52.258 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-02-05T07:00:52.293 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-02-07T07:00:14.421369+0000"' 2024-02-05T07:00:52.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-02-07T07:00:14.421369+0000 2024-02-05T07:00:52.293 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-02-07T07:00:14.421369+0000 2024-02-05T07:00:52.294 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T07:00:52.296 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2024-02-05T07:00:52.297 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-02-07 = 2024-02-07 2024-02-05T07:00:52.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-07T07:00:14.421369+0000",' '"deadline":' '"2024-02-19T07:00:14.421369+0000",' '"forced":' false '}' ']' 2024-02-05T07:00:52.298 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-02-05T07:00:52.338 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-02-19T07:00:14.421369+0000"' 2024-02-05T07:00:52.338 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-02-19T07:00:14.421369+0000 2024-02-05T07:00:52.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-02-19T07:00:14.421369+0000 2024-02-05T07:00:52.339 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T07:00:52.341 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 week' 2024-02-05T07:00:52.343 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-02-19 = 2024-02-19 2024-02-05T07:00:52.343 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:197: TEST_interval_changes: expr 86400 '*' 3 2024-02-05T07:00:52.344 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:197: TEST_interval_changes: ceph osd pool set test scrub_min_interval 259200 2024-02-05T07:00:52.753 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 1 scrub_min_interval to 259200 2024-02-05T07:00:52.773 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:198: TEST_interval_changes: sleep 15 2024-02-05T07:01:07.775 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:199: TEST_interval_changes: check_dump_scrubs 1 '3 days' '2 week' 2024-02-05T07:01:07.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-02-05T07:01:07.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=3 days' 2024-02-05T07:01:07.776 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=2 week' 2024-02-05T07:01:07.777 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-02-05T07:01:07.777 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T07:01:07.777 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T07:01:07.778 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T07:01:07.778 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T07:01:07.778 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T07:01:07.778 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.545011/ceph-osd.1.asok 2024-02-05T07:01:07.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-02-05T07:01:07.779 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.545011/ceph-osd.1.asok dump_scrubs 2024-02-05T07:01:07.907 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-02-05T07:01:07.907 INFO:tasks.workunit.client.0.smithi148.stderr: { 2024-02-05T07:01:07.907 INFO:tasks.workunit.client.0.smithi148.stderr: "pgid": "1.0", 2024-02-05T07:01:07.907 INFO:tasks.workunit.client.0.smithi148.stderr: "sched_time": "2024-02-08T07:00:14.421369+0000", 2024-02-05T07:01:07.907 INFO:tasks.workunit.client.0.smithi148.stderr: "deadline": "2024-02-19T07:00:14.421369+0000", 2024-02-05T07:01:07.907 INFO:tasks.workunit.client.0.smithi148.stderr: "forced": false 2024-02-05T07:01:07.907 INFO:tasks.workunit.client.0.smithi148.stderr: } 2024-02-05T07:01:07.907 INFO:tasks.workunit.client.0.smithi148.stderr:]' 2024-02-05T07:01:07.908 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-08T07:00:14.421369+0000",' '"deadline":' '"2024-02-19T07:00:14.421369+0000",' '"forced":' false '}' ']' 2024-02-05T07:01:07.908 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-02-05T07:01:07.944 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-02-08T07:00:14.421369+0000"' 2024-02-05T07:01:07.944 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-02-08T07:00:14.421369+0000 2024-02-05T07:01:07.945 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-02-08T07:00:14.421369+0000 2024-02-05T07:01:07.945 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T07:01:07.947 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2024-02-05T07:01:07.948 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-02-08 = 2024-02-08 2024-02-05T07:01:07.949 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-08T07:00:14.421369+0000",' '"deadline":' '"2024-02-19T07:00:14.421369+0000",' '"forced":' false '}' ']' 2024-02-05T07:01:07.949 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-02-05T07:01:07.984 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-02-19T07:00:14.421369+0000"' 2024-02-05T07:01:07.984 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-02-19T07:00:14.421369+0000 2024-02-05T07:01:07.984 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-02-19T07:00:14.421369+0000 2024-02-05T07:01:07.985 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T07:01:07.986 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 week' 2024-02-05T07:01:07.987 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-02-19 = 2024-02-19 2024-02-05T07:01:07.987 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:202: TEST_interval_changes: expr 604800 '*' 3 2024-02-05T07:01:07.988 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:202: TEST_interval_changes: ceph osd pool set test scrub_max_interval 1814400 2024-02-05T07:01:08.385 INFO:tasks.workunit.client.0.smithi148.stderr:set pool 1 scrub_max_interval to 1814400 2024-02-05T07:01:08.404 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:203: TEST_interval_changes: sleep 15 2024-02-05T07:01:23.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:204: TEST_interval_changes: check_dump_scrubs 1 '3 days' '3 week' 2024-02-05T07:01:23.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-02-05T07:01:23.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=3 days' 2024-02-05T07:01:23.407 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=3 week' 2024-02-05T07:01:23.408 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-02-05T07:01:23.408 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T07:01:23.408 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T07:01:23.409 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T07:01:23.409 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T07:01:23.409 INFO:tasks.workunit.client.0.smithi148.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T07:01:23.409 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.545011/ceph-osd.1.asok 2024-02-05T07:01:23.409 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-02-05T07:01:23.410 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.545011/ceph-osd.1.asok dump_scrubs 2024-02-05T07:01:23.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-02-05T07:01:23.541 INFO:tasks.workunit.client.0.smithi148.stderr: { 2024-02-05T07:01:23.541 INFO:tasks.workunit.client.0.smithi148.stderr: "pgid": "1.0", 2024-02-05T07:01:23.542 INFO:tasks.workunit.client.0.smithi148.stderr: "sched_time": "2024-02-08T07:00:14.421369+0000", 2024-02-05T07:01:23.542 INFO:tasks.workunit.client.0.smithi148.stderr: "deadline": "2024-02-26T07:00:14.421369+0000", 2024-02-05T07:01:23.542 INFO:tasks.workunit.client.0.smithi148.stderr: "forced": false 2024-02-05T07:01:23.542 INFO:tasks.workunit.client.0.smithi148.stderr: } 2024-02-05T07:01:23.542 INFO:tasks.workunit.client.0.smithi148.stderr:]' 2024-02-05T07:01:23.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-08T07:00:14.421369+0000",' '"deadline":' '"2024-02-26T07:00:14.421369+0000",' '"forced":' false '}' ']' 2024-02-05T07:01:23.543 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-02-05T07:01:23.577 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-02-08T07:00:14.421369+0000"' 2024-02-05T07:01:23.577 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-02-08T07:00:14.421369+0000 2024-02-05T07:01:23.578 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T07:01:23.578 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-02-08T07:00:14.421369+0000 2024-02-05T07:01:23.581 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2024-02-05T07:01:23.582 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-02-08 = 2024-02-08 2024-02-05T07:01:23.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-08T07:00:14.421369+0000",' '"deadline":' '"2024-02-26T07:00:14.421369+0000",' '"forced":' false '}' ']' 2024-02-05T07:01:23.583 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-02-05T07:01:23.613 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-02-26T07:00:14.421369+0000"' 2024-02-05T07:01:23.613 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-02-26T07:00:14.421369+0000 2024-02-05T07:01:23.614 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-02-26T07:00:14.421369+0000 2024-02-05T07:01:23.614 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T07:01:23.615 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 week' 2024-02-05T07:01:23.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-02-26 = 2024-02-26 2024-02-05T07:01:23.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:205: TEST_interval_changes: perf_counters td/osd-scrub-test 2 2024-02-05T07:01:23.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-02-05T07:01:23.617 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=2 2024-02-05T07:01:23.618 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 2 - 1 2024-02-05T07:01:23.618 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 1 2024-02-05T07:01:23.618 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T07:01:23.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-02-05T07:01:23.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T07:01:23.760 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T07:01:23.760 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_ec": [ 2024-02-05T07:01:23.761 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T07:01:23.761 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T07:01:23.761 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T07:01:23.761 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T07:01:23.761 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.761 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T07:01:23.761 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T07:01:23.761 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T07:01:23.761 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T07:01:23.761 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T07:01:23.762 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.762 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.762 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.762 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.762 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T07:01:23.762 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T07:01:23.762 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.762 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.762 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.762 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.763 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T07:01:23.763 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T07:01:23.763 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T07:01:23.763 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T07:01:23.763 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T07:01:23.763 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T07:01:23.763 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T07:01:23.763 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.763 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.763 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.764 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.764 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T07:01:23.764 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T07:01:23.764 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T07:01:23.764 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T07:01:23.764 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T07:01:23.764 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.764 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.764 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.765 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.765 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T07:01:23.765 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.765 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.765 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T07:01:23.765 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_repl": [ 2024-02-05T07:01:23.765 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T07:01:23.765 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T07:01:23.765 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T07:01:23.766 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T07:01:23.766 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.766 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T07:01:23.766 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T07:01:23.766 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T07:01:23.766 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T07:01:23.766 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T07:01:23.766 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.766 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.766 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.767 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.767 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T07:01:23.767 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T07:01:23.767 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.767 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.767 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.767 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.767 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T07:01:23.767 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T07:01:23.767 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T07:01:23.767 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T07:01:23.767 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T07:01:23.767 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T07:01:23.767 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T07:01:23.768 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.768 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.768 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.768 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.768 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T07:01:23.768 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T07:01:23.768 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T07:01:23.768 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T07:01:23.768 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T07:01:23.768 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.768 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.769 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.769 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.769 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T07:01:23.769 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.769 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.769 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T07:01:23.769 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_ec": [ 2024-02-05T07:01:23.769 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T07:01:23.769 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T07:01:23.769 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T07:01:23.769 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T07:01:23.769 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.769 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T07:01:23.770 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T07:01:23.770 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T07:01:23.770 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T07:01:23.770 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T07:01:23.770 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.770 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.770 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.770 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.770 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T07:01:23.770 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T07:01:23.770 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.770 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.770 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.771 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.771 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T07:01:23.771 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T07:01:23.771 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T07:01:23.771 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T07:01:23.771 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T07:01:23.771 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T07:01:23.771 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T07:01:23.771 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.771 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.771 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.771 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.772 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T07:01:23.772 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T07:01:23.772 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T07:01:23.772 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T07:01:23.772 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T07:01:23.772 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.772 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.772 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.772 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.772 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T07:01:23.773 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.773 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.773 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T07:01:23.773 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_repl": [ 2024-02-05T07:01:23.773 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T07:01:23.773 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T07:01:23.773 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T07:01:23.773 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T07:01:23.773 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.773 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T07:01:23.773 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T07:01:23.773 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T07:01:23.773 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T07:01:23.773 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T07:01:23.774 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.774 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.774 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.774 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.774 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T07:01:23.774 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T07:01:23.774 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.774 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.774 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.774 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.774 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T07:01:23.774 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T07:01:23.774 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T07:01:23.774 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T07:01:23.775 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T07:01:23.775 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T07:01:23.775 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T07:01:23.775 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.775 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.775 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.775 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.776 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T07:01:23.776 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T07:01:23.776 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T07:01:23.776 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T07:01:23.776 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T07:01:23.776 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.776 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.776 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.776 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.776 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T07:01:23.776 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.776 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.776 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T07:01:23.776 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T07:01:23.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T07:01:23.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-02-05T07:01:23.777 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T07:01:23.922 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T07:01:23.922 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_ec": [ 2024-02-05T07:01:23.922 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T07:01:23.922 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T07:01:23.922 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T07:01:23.922 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.923 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.924 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.924 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T07:01:23.924 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T07:01:23.924 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T07:01:23.924 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T07:01:23.924 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T07:01:23.924 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T07:01:23.924 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T07:01:23.924 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.924 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.924 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.924 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.924 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_dp_repl": [ 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T07:01:23.925 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T07:01:23.926 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "deep", 2024-02-05T07:01:23.926 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T07:01:23.926 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.926 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T07:01:23.926 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T07:01:23.926 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T07:01:23.926 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T07:01:23.926 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T07:01:23.926 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.926 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.926 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.926 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.927 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T07:01:23.927 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T07:01:23.927 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.927 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.927 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.927 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.927 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T07:01:23.927 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T07:01:23.927 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T07:01:23.927 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T07:01:23.927 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T07:01:23.928 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T07:01:23.928 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T07:01:23.928 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.928 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.928 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.928 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.928 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T07:01:23.928 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T07:01:23.928 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T07:01:23.928 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T07:01:23.929 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T07:01:23.929 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.929 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.929 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.929 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.929 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T07:01:23.929 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.929 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.929 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T07:01:23.929 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_ec": [ 2024-02-05T07:01:23.930 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T07:01:23.930 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T07:01:23.930 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T07:01:23.930 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "ec" 2024-02-05T07:01:23.930 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.930 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T07:01:23.930 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T07:01:23.930 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T07:01:23.930 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T07:01:23.930 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T07:01:23.930 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.930 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.931 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.931 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.931 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T07:01:23.931 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T07:01:23.931 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.931 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.931 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.931 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.931 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T07:01:23.931 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T07:01:23.931 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T07:01:23.932 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T07:01:23.932 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T07:01:23.932 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T07:01:23.932 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T07:01:23.932 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.932 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.932 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.932 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.932 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T07:01:23.932 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T07:01:23.932 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T07:01:23.932 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: ], 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: "osd_scrub_sh_repl": [ 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: { 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: "labels": { 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: "level": "shallow", 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: "pooltype": "replicated" 2024-02-05T07:01:23.933 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: "counters": { 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_started": 0, 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs": 0, 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_scrubs_elapsed": { 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs": 0, 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_scrubs_elapsed": { 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.934 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.935 INFO:tasks.workunit.client.0.smithi148.stdout: "preemptions": 0, 2024-02-05T07:01:23.935 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_selected": 0, 2024-02-05T07:01:23.935 INFO:tasks.workunit.client.0.smithi148.stdout: "chunk_busy": 0, 2024-02-05T07:01:23.935 INFO:tasks.workunit.client.0.smithi148.stdout: "locked_object": 0, 2024-02-05T07:01:23.935 INFO:tasks.workunit.client.0.smithi148.stdout: "write_blocked_by_scrub": 0, 2024-02-05T07:01:23.935 INFO:tasks.workunit.client.0.smithi148.stdout: "scrub_reservations_completed": 0, 2024-02-05T07:01:23.935 INFO:tasks.workunit.client.0.smithi148.stdout: "successful_reservations_elapsed": { 2024-02-05T07:01:23.936 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.936 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.936 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.936 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.936 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_aborted": 0, 2024-02-05T07:01:23.936 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_timed_out": 0, 2024-02-05T07:01:23.936 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_failure": 0, 2024-02-05T07:01:23.936 INFO:tasks.workunit.client.0.smithi148.stdout: "reservation_process_skipped": 0, 2024-02-05T07:01:23.936 INFO:tasks.workunit.client.0.smithi148.stdout: "failed_reservations_elapsed": { 2024-02-05T07:01:23.936 INFO:tasks.workunit.client.0.smithi148.stdout: "avgcount": 0, 2024-02-05T07:01:23.936 INFO:tasks.workunit.client.0.smithi148.stdout: "sum": 0, 2024-02-05T07:01:23.937 INFO:tasks.workunit.client.0.smithi148.stdout: "avgtime": 0 2024-02-05T07:01:23.937 INFO:tasks.workunit.client.0.smithi148.stdout: }, 2024-02-05T07:01:23.937 INFO:tasks.workunit.client.0.smithi148.stdout: "replicas_in_reservation": 0 2024-02-05T07:01:23.937 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.937 INFO:tasks.workunit.client.0.smithi148.stdout: } 2024-02-05T07:01:23.937 INFO:tasks.workunit.client.0.smithi148.stdout: ] 2024-02-05T07:01:23.937 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T07:01:23.938 INFO:tasks.workunit.client.0.smithi148.stdout:-------------- Teardown Test TEST_interval_changes ------------------ 2024-02-05T07:01:23.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_interval_changes ------------------' 2024-02-05T07:01:23.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-02-05T07:01:23.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T07:01:23.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T07:01:23.939 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T07:01:23.940 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T07:01:23.940 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T07:01:23.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T07:01:23.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T07:01:23.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T07:01:24.044 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T07:01:24.045 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T07:01:24.045 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T07:01:24.046 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T07:01:24.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T07:01:24.047 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T07:01:24.047 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T07:01:24.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T07:01:24.048 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T07:01:24.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T07:01:24.049 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T07:01:24.049 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T07:01:24.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T07:01:24.050 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T07:01:24.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T07:01:24.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T07:01:24.064 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T07:01:24.064 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.545011 2024-02-05T07:01:24.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T07:01:24.065 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T07:01:24.066 INFO:tasks.workunit.client.0.smithi148.stdout:-------------- Complete Test TEST_interval_changes ------------------ 2024-02-05T07:01:24.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_interval_changes ------------------' 2024-02-05T07:01:24.066 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-02-05T07:01:24.066 INFO:tasks.workunit.client.0.smithi148.stdout:-------------- Prepare Test TEST_just_deep_scrubs ------------------- 2024-02-05T07:01:24.067 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_just_deep_scrubs -------------------' 2024-02-05T07:01:24.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-02-05T07:01:24.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-02-05T07:01:24.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-02-05T07:01:24.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T07:01:24.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T07:01:24.068 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T07:01:24.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T07:01:24.069 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T07:01:24.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T07:01:24.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T07:01:24.069 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T07:01:24.070 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T07:01:24.070 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T07:01:24.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T07:01:24.071 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T07:01:24.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T07:01:24.071 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T07:01:24.072 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T07:01:24.072 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T07:01:24.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T07:01:24.073 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T07:01:24.074 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T07:01:24.074 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T07:01:24.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T07:01:24.075 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T07:01:24.076 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T07:01:24.076 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T07:01:24.076 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T07:01:24.076 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.545011 2024-02-05T07:01:24.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T07:01:24.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T07:01:24.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-02-05T07:01:24.079 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T07:01:24.079 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T07:01:24.079 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T07:01:24.079 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.545011 2024-02-05T07:01:24.080 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T07:01:24.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T07:01:24.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T07:01:24.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-02-05T07:01:24.081 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_just_deep_scrubs -----------------------' 2024-02-05T07:01:24.081 INFO:tasks.workunit.client.0.smithi148.stdout:-------------- Run Test TEST_just_deep_scrubs ----------------------- 2024-02-05T07:01:24.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_just_deep_scrubs td/osd-scrub-test 2024-02-05T07:01:24.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:467: TEST_just_deep_scrubs: local dir=td/osd-scrub-test 2024-02-05T07:01:24.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:468: TEST_just_deep_scrubs: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-02-05T07:01:24.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:468: TEST_just_deep_scrubs: local -A cluster_conf 2024-02-05T07:01:24.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:474: TEST_just_deep_scrubs: standard_scrub_cluster td/osd-scrub-test cluster_conf 2024-02-05T07:01:24.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-test 2024-02-05T07:01:24.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-02-05T07:01:24.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-02-05T07:01:24.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-02-05T07:01:24.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-02-05T07:01:24.083 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-02-05T07:01:24.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-02-05T07:01:24.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-02-05T07:01:24.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-02-05T07:01:24.084 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-02-05T07:01:25.811 INFO:tasks.workunit.client.0.smithi148.stdout:add osd0 b7314347-7297-48c4-88ae-f09f21b43adc 2024-02-05T07:01:26.076 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T07:01:26.129 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:01:26.123+0000 7fb7b6862640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:01:26.131 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:01:26.123+0000 7fb7b6862640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:01:26.132 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:01:26.127+0000 7fb7b6862640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:01:26.133 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:01:26.127+0000 7fb7b6862640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T07:01:26.133 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:01:26.127+0000 7fb7b6862640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T07:01:26.133 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:01:26.127+0000 7fb7b6862640 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T07:01:26.219 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:01:26.211+0000 7fb7b6862640 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-02-05T07:01:32.282 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd0 key to auth repository 2024-02-05T07:01:32.732 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.0 2024-02-05T07:01:32.770 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:01:32.763+0000 7fb3669f3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:01:32.809 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:01:32.803+0000 7fb3669f3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:01:32.810 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:01:32.803+0000 7fb3669f3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:01:33.035 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T07:01:33.361 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:01:33.355+0000 7fb3669f3640 -1 Falling back to public interface 2024-02-05T07:01:34.379 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T07:01:35.683 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T07:01:36.019 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:01:36.015+0000 7fb3669f3640 -1 osd.0 0 log_to_monitors true 2024-02-05T07:01:36.995 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T07:01:38.314 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T07:01:39.685 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T07:01:41.139 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T07:01:42.455 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T07:01:43.855 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T07:01:45.168 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T07:01:46.482 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T07:01:47.845 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T07:01:49.159 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T07:01:50.867 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T07:01:52.359 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T07:01:53.738 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T07:01:55.288 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T07:01:56.861 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T07:01:58.518 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T07:01:59.997 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T07:02:00.369 INFO:tasks.workunit.client.0.smithi148.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2431507164,v1:127.0.0.1:6803/2431507164] [v2:127.0.0.1:6804/2431507164,v1:127.0.0.1:6805/2431507164] exists,up b7314347-7297-48c4-88ae-f09f21b43adc 2024-02-05T07:02:00.375 INFO:tasks.workunit.client.0.smithi148.stdout:add osd1 cb79273d-63f8-45fc-af1c-30caa0ec9e56 2024-02-05T07:02:00.760 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T07:02:00.817 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:00.810+0000 7f2ae904f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:02:00.844 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:00.838+0000 7f2ae904f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:02:00.846 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:00.838+0000 7f2ae904f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:02:00.846 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:00.838+0000 7f2ae904f640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T07:02:00.846 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:00.838+0000 7f2ae904f640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T07:02:00.846 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:00.838+0000 7f2ae904f640 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T07:02:00.917 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:00.914+0000 7f2ae904f640 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-02-05T07:02:07.316 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd1 key to auth repository 2024-02-05T07:02:07.701 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.1 2024-02-05T07:02:07.735 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:07.730+0000 7f955c295640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:02:07.758 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:07.754+0000 7f955c295640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:02:07.759 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:07.758+0000 7f955c295640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:02:08.028 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T07:02:08.310 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:08.306+0000 7f955c295640 -1 Falling back to public interface 2024-02-05T07:02:09.365 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T07:02:10.670 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T07:02:10.984 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:10.982+0000 7f955c295640 -1 osd.1 0 log_to_monitors true 2024-02-05T07:02:11.994 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T07:02:13.336 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T07:02:14.668 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T07:02:16.215 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T07:02:17.519 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T07:02:18.846 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T07:02:20.288 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T07:02:21.592 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T07:02:23.003 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T07:02:24.334 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T07:02:25.648 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T07:02:27.622 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T07:02:29.140 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T07:02:30.575 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T07:02:32.360 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T07:02:33.828 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T07:02:35.340 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T07:02:36.655 INFO:tasks.workunit.client.0.smithi148.stdout:20 2024-02-05T07:02:36.985 INFO:tasks.workunit.client.0.smithi148.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3257310796,v1:127.0.0.1:6811/3257310796] [v2:127.0.0.1:6812/3257310796,v1:127.0.0.1:6813/3257310796] exists,up cb79273d-63f8-45fc-af1c-30caa0ec9e56 2024-02-05T07:02:36.990 INFO:tasks.workunit.client.0.smithi148.stdout:add osd2 8200b945-bb6b-482e-884e-c2a2f75a0cd6 2024-02-05T07:02:37.519 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T07:02:37.581 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:37.577+0000 7f9787e46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:02:37.612 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:37.609+0000 7f9787e46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:02:37.614 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:37.609+0000 7f9787e46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:02:37.615 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:37.609+0000 7f9787e46640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T07:02:37.615 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:37.609+0000 7f9787e46640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T07:02:37.615 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:37.609+0000 7f9787e46640 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T07:02:37.686 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:37.681+0000 7f9787e46640 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-02-05T07:02:44.213 INFO:tasks.workunit.client.0.smithi148.stdout:adding osd2 key to auth repository 2024-02-05T07:02:44.593 INFO:tasks.workunit.client.0.smithi148.stdout:start osd.2 2024-02-05T07:02:44.628 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:44.625+0000 7f561ff42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:02:44.652 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:44.649+0000 7f561ff42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:02:44.654 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:44.649+0000 7f561ff42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T07:02:44.921 INFO:tasks.workunit.client.0.smithi148.stdout:0 2024-02-05T07:02:45.202 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:45.197+0000 7f561ff42640 -1 Falling back to public interface 2024-02-05T07:02:46.235 INFO:tasks.workunit.client.0.smithi148.stdout:1 2024-02-05T07:02:47.537 INFO:tasks.workunit.client.0.smithi148.stdout:2 2024-02-05T07:02:47.756 INFO:tasks.workunit.client.0.smithi148.stderr:2024-02-05T07:02:47.753+0000 7f561ff42640 -1 osd.2 0 log_to_monitors true 2024-02-05T07:02:48.861 INFO:tasks.workunit.client.0.smithi148.stdout:3 2024-02-05T07:02:50.289 INFO:tasks.workunit.client.0.smithi148.stdout:4 2024-02-05T07:02:51.650 INFO:tasks.workunit.client.0.smithi148.stdout:5 2024-02-05T07:02:53.002 INFO:tasks.workunit.client.0.smithi148.stdout:6 2024-02-05T07:02:54.443 INFO:tasks.workunit.client.0.smithi148.stdout:7 2024-02-05T07:02:55.888 INFO:tasks.workunit.client.0.smithi148.stdout:8 2024-02-05T07:02:57.247 INFO:tasks.workunit.client.0.smithi148.stdout:9 2024-02-05T07:02:58.674 INFO:tasks.workunit.client.0.smithi148.stdout:10 2024-02-05T07:02:59.988 INFO:tasks.workunit.client.0.smithi148.stdout:11 2024-02-05T07:03:01.325 INFO:tasks.workunit.client.0.smithi148.stdout:12 2024-02-05T07:03:03.151 INFO:tasks.workunit.client.0.smithi148.stdout:13 2024-02-05T07:03:04.605 INFO:tasks.workunit.client.0.smithi148.stdout:14 2024-02-05T07:03:06.335 INFO:tasks.workunit.client.0.smithi148.stdout:15 2024-02-05T07:03:08.158 INFO:tasks.workunit.client.0.smithi148.stdout:16 2024-02-05T07:03:09.885 INFO:tasks.workunit.client.0.smithi148.stdout:17 2024-02-05T07:03:11.446 INFO:tasks.workunit.client.0.smithi148.stdout:18 2024-02-05T07:03:12.775 INFO:tasks.workunit.client.0.smithi148.stdout:19 2024-02-05T07:03:13.078 INFO:tasks.workunit.client.0.smithi148.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1873398175,v1:127.0.0.1:6819/1873398175] [v2:127.0.0.1:6820/1873398175,v1:127.0.0.1:6821/1873398175] exists,up 8200b945-bb6b-482e-884e-c2a2f75a0cd6 2024-02-05T07:03:13.513 INFO:tasks.workunit.client.0.smithi148.stderr:pool 'test' created 2024-02-05T07:03:15.568 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.0 seq 21474836500 2024-02-05T07:03:17.208 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.1 seq 42949672971 2024-02-05T07:03:17.549 INFO:tasks.workunit.client.0.smithi148.stdout:waiting osd.2 seq 64424509445 2024-02-05T07:03:20.307 INFO:tasks.workunit.client.0.smithi148.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-02-05T07:03:20.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:475: TEST_just_deep_scrubs: local poolid=1 2024-02-05T07:03:20.307 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:476: TEST_just_deep_scrubs: local poolname=test 2024-02-05T07:03:20.308 INFO:tasks.workunit.client.0.smithi148.stdout:Pool: test : 1 2024-02-05T07:03:20.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: echo 'Pool: test : 1' 2024-02-05T07:03:20.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: TESTDATA=testdata.545011 2024-02-05T07:03:20.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:480: TEST_just_deep_scrubs: local objects=15 2024-02-05T07:03:20.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:481: TEST_just_deep_scrubs: dd if=/dev/urandom of=testdata.545011 bs=1032 count=1 2024-02-05T07:03:20.309 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records in 2024-02-05T07:03:20.309 INFO:tasks.workunit.client.0.smithi148.stderr:1+0 records out 2024-02-05T07:03:20.309 INFO:tasks.workunit.client.0.smithi148.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000112374 s, 9.2 MB/s 2024-02-05T07:03:20.309 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: seq 1 15 2024-02-05T07:03:20.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:20.310 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj1 testdata.545011 2024-02-05T07:03:20.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:20.478 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj2 testdata.545011 2024-02-05T07:03:20.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:20.629 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj3 testdata.545011 2024-02-05T07:03:20.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:20.780 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj4 testdata.545011 2024-02-05T07:03:20.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:20.940 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj5 testdata.545011 2024-02-05T07:03:21.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:21.142 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj6 testdata.545011 2024-02-05T07:03:21.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:21.309 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj7 testdata.545011 2024-02-05T07:03:21.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:21.468 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj8 testdata.545011 2024-02-05T07:03:21.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:21.619 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj9 testdata.545011 2024-02-05T07:03:21.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:21.786 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj10 testdata.545011 2024-02-05T07:03:21.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:21.928 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj11 testdata.545011 2024-02-05T07:03:22.087 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:22.088 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj12 testdata.545011 2024-02-05T07:03:22.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:22.239 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj13 testdata.545011 2024-02-05T07:03:22.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:22.390 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj14 testdata.545011 2024-02-05T07:03:22.540 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T07:03:22.541 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj15 testdata.545011 2024-02-05T07:03:22.700 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rm -f testdata.545011 2024-02-05T07:03:22.701 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:491: TEST_just_deep_scrubs: ceph osd set noscrub 2024-02-05T07:03:23.172 INFO:tasks.workunit.client.0.smithi148.stderr:noscrub is set 2024-02-05T07:03:23.198 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:492: TEST_just_deep_scrubs: ceph osd set nodeep-scrub 2024-02-05T07:03:23.742 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is set 2024-02-05T07:03:23.765 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:493: TEST_just_deep_scrubs: sleep 6 2024-02-05T07:03:29.767 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:494: TEST_just_deep_scrubs: date -Ins 2024-02-05T07:03:29.768 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:494: TEST_just_deep_scrubs: local now_is=2024-02-05T07:03:29,767435905+00:00 2024-02-05T07:03:29.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:495: TEST_just_deep_scrubs: declare -A sched_data 2024-02-05T07:03:29.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:496: TEST_just_deep_scrubs: local pgid=1.2 2024-02-05T07:03:29.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:499: TEST_just_deep_scrubs: set_query_debug 1.2 2024-02-05T07:03:29.769 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:294: set_query_debug: local pgid=1.2 2024-02-05T07:03:29.770 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: ceph pg dump pgs_brief 2024-02-05T07:03:29.770 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: awk -v 'pg=^1.2' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-02-05T07:03:30.060 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs_brief 2024-02-05T07:03:30.077 INFO:tasks.workunit.client.0.smithi148.stdout:Setting scrub debug data. Primary for 1.2 is 0 2024-02-05T07:03:30.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: local prim_osd=0 2024-02-05T07:03:30.077 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: echo 'Setting scrub debug data. Primary for 1.2 is 0' 2024-02-05T07:03:30.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: get_asok_path osd.0 2024-02-05T07:03:30.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T07:03:30.078 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T07:03:30.079 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T07:03:30.079 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T07:03:30.079 INFO:tasks.workunit.client.0.smithi148.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T07:03:30.079 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.545011/ceph-osd.0.asok 2024-02-05T07:03:30.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: CEPH_ARGS= 2024-02-05T07:03:30.080 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.545011/ceph-osd.0.asok scrubdebug 1.2 set sessions 2024-02-05T07:03:30.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:501: TEST_just_deep_scrubs: extract_published_sch 1.2 2024-02-05T07:03:29,767435905+00:00 2024-02-05T07:03:29,767435905+00:00 sched_data 2024-02-05T07:03:30.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2024-02-05T07:03:30.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2024-02-05T07:03:30.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-02-05T07:03:29,767435905+00:00 2024-02-05T07:03:30.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2024-02-05T07:03:29,767435905+00:00 2024-02-05T07:03:30.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=1 2024-02-05T07:03:30.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-02-05T07:03:30.210 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-02-05T07:03:30.502 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T07:03:30.704 INFO:tasks.workunit.client.0.smithi148.stdout:{"success":true}( 2024-02-05T07:03:30.704 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=22 2024-02-05T07:03:30.704 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=15 2024-02-05T07:03:30.704 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T07:03:30.704 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T07:03:30.704 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T07:03:13.450' 2024-02-05T07:03:30.704 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=0 2024-02-05T07:03:30.704 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T07:03:13.450260+0000' 2024-02-05T07:03:30.704 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='0x0' 2024-02-05T07:03:30.704 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T07:03:30.705 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T07:03:30.705 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=9747328 2024-02-05T07:03:30.705 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T07:03:30.705 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T07:03:30.705 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=0 2024-02-05T07:03:30.705 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T07:03:30.705 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T07:03:13.450260+0000' 2024-02-05T07:03:30.705 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T07:03:30.705 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T07:03:30.705 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=21 2024-02-05T07:03:30.705 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=14 2024-02-05T07:03:30.705 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T07:03:30.737 INFO:tasks.workunit.client.0.smithi148.stdout:test counter @ start: 9747328 2024-02-05T07:03:30.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:502: TEST_just_deep_scrubs: local saved_last_stamp=2024-02-05T07:03:13.450260+0000 2024-02-05T07:03:30.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:503: TEST_just_deep_scrubs: local dbg_counter_at_start=9747328 2024-02-05T07:03:30.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:504: TEST_just_deep_scrubs: echo 'test counter @ start: 9747328' 2024-02-05T07:03:30.737 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:506: TEST_just_deep_scrubs: ceph tell 1.2 schedule-deep-scrub 2024-02-05T07:03:30.867 INFO:tasks.workunit.client.0.smithi148.stdout:{ 2024-02-05T07:03:30.867 INFO:tasks.workunit.client.0.smithi148.stdout: "deep": true, 2024-02-05T07:03:30.867 INFO:tasks.workunit.client.0.smithi148.stdout: "must": false, 2024-02-05T07:03:30.867 INFO:tasks.workunit.client.0.smithi148.stdout: "stamp": "2024-01-22T07:01:50.865970+0000" 2024-02-05T07:03:30.867 INFO:tasks.workunit.client.0.smithi148.stdout:} 2024-02-05T07:03:30.881 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:508: TEST_just_deep_scrubs: sleep 5 2024-02-05T07:03:35.883 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:509: TEST_just_deep_scrubs: declare -A sc_data_2 2024-02-05T07:03:35.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:510: TEST_just_deep_scrubs: extract_published_sch 1.2 2024-02-05T07:03:29,767435905+00:00 2024-02-05T07:03:29,767435905+00:00 sc_data_2 2024-02-05T07:03:35.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2024-02-05T07:03:35.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sc_data_2 2024-02-05T07:03:35.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-02-05T07:03:29,767435905+00:00 2024-02-05T07:03:35.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2024-02-05T07:03:29,767435905+00:00 2024-02-05T07:03:35.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=1 2024-02-05T07:03:35.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-02-05T07:03:35.884 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-02-05T07:03:36.180 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T07:03:36.375 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T07:03:36.375 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=22 2024-02-05T07:03:36.375 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=19 2024-02-05T07:03:36.375 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T07:03:36.376 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='queued for deep scrub' 2024-02-05T07:03:36.376 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='0' 2024-02-05T07:03:36.376 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=0 2024-02-05T07:03:36.376 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-01-22T07:01:50.865970+0000' 2024-02-05T07:03:36.376 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='0x0' 2024-02-05T07:03:36.376 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=false 2024-02-05T07:03:36.376 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=false 2024-02-05T07:03:36.376 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=9747328 2024-02-05T07:03:36.376 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T07:03:36.376 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T07:03:36.376 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=0 2024-02-05T07:03:36.376 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T07:03:36.376 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T07:03:13.450260+0000' 2024-02-05T07:03:36.377 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T07:03:36.377 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T07:03:36.377 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=21 2024-02-05T07:03:36.377 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=14 2024-02-05T07:03:36.377 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T07:03:36.405 INFO:tasks.workunit.client.0.smithi148.stdout:test counter @ should show no change: 9747328 2024-02-05T07:03:36.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:511: TEST_just_deep_scrubs: echo 'test counter @ should show no change: ' 9747328 2024-02-05T07:03:36.405 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:512: TEST_just_deep_scrubs: (( 0 == 0 )) 2024-02-05T07:03:36.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:513: TEST_just_deep_scrubs: (( 9747328 == 9747328 )) 2024-02-05T07:03:36.406 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:516: TEST_just_deep_scrubs: ceph osd unset nodeep-scrub 2024-02-05T07:03:36.800 INFO:tasks.workunit.client.0.smithi148.stderr:nodeep-scrub is unset 2024-02-05T07:03:36.818 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:517: TEST_just_deep_scrubs: sleep 5 2024-02-05T07:03:41.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:518: TEST_just_deep_scrubs: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2024-02-05T07:03:41.821 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:518: TEST_just_deep_scrubs: declare -A expct_qry_duration 2024-02-05T07:03:41.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:519: TEST_just_deep_scrubs: sc_data_2=() 2024-02-05T07:03:41.822 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:520: TEST_just_deep_scrubs: echo 'test counter @ should be higher than before the unset: ' 2024-02-05T07:03:41.822 INFO:tasks.workunit.client.0.smithi148.stdout:test counter @ should be higher than before the unset: 2024-02-05T07:03:41.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:521: TEST_just_deep_scrubs: wait_any_cond 1.2 10 2024-02-05T07:03:13.450260+0000 expct_qry_duration 'WaitingAfterScrub ' sc_data_2 2024-02-05T07:03:41.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.2 2024-02-05T07:03:41.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-02-05T07:03:41.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-02-05T07:03:13.450260+0000 2024-02-05T07:03:41.823 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_qry_duration 2024-02-05T07:03:41.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sc_data_2 2024-02-05T07:03:41.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-02-05T07:03:41.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-02-05T07:03:41.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-02-05T07:03:41.824 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-02-05T07:03:41.824 INFO:tasks.workunit.client.0.smithi148.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.2 dt:2024-02-05T07:03:13.450260+0000 (10 retries) 2024-02-05T07:03:42.623 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T07:03:42.820 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T07:03:42.821 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=23 2024-02-05T07:03:42.821 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=28 2024-02-05T07:03:42.821 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T07:03:42.821 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T07:03:42.821 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T07:03:38.516' 2024-02-05T07:03:42.821 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=0 2024-02-05T07:03:42.821 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T07:03:38.516618+0000' 2024-02-05T07:03:42.821 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='0x0' 2024-02-05T07:03:42.821 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T07:03:42.821 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T07:03:42.821 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=9747331 2024-02-05T07:03:42.821 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T07:03:42.822 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T07:03:42.822 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=0 2024-02-05T07:03:42.822 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T07:03:42.822 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T07:03:38.516618+0000' 2024-02-05T07:03:42.822 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T07:03:42.822 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T07:03:42.822 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=23 2024-02-05T07:03:42.822 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=27 2024-02-05T07:03:42.822 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T07:03:42.851 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 1 ~ false / 28 / 27 / true / 2024-02-05T07:03:38.516618+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T07:03:42.852 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T07:03:43.646 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T07:03:43.843 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T07:03:43.843 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=23 2024-02-05T07:03:43.843 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=28 2024-02-05T07:03:43.843 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T07:03:43.843 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T07:03:43.844 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T07:03:38.516' 2024-02-05T07:03:43.844 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=0 2024-02-05T07:03:43.844 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T07:03:38.516618+0000' 2024-02-05T07:03:43.844 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='0x0' 2024-02-05T07:03:43.844 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T07:03:43.844 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T07:03:43.844 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=9747331 2024-02-05T07:03:43.844 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T07:03:43.844 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T07:03:43.844 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=0 2024-02-05T07:03:43.844 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T07:03:43.844 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T07:03:38.516618+0000' 2024-02-05T07:03:43.844 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T07:03:43.844 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T07:03:43.845 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=23 2024-02-05T07:03:43.845 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=27 2024-02-05T07:03:43.845 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T07:03:43.876 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 2 ~ false / 28 / 27 / true / 2024-02-05T07:03:38.516618+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T07:03:43.876 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T07:03:44.675 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T07:03:44.871 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T07:03:44.871 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=23 2024-02-05T07:03:44.871 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=28 2024-02-05T07:03:44.871 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T07:03:44.871 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T07:03:44.871 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T07:03:38.516' 2024-02-05T07:03:44.871 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=0 2024-02-05T07:03:44.871 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T07:03:38.516618+0000' 2024-02-05T07:03:44.871 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='0x0' 2024-02-05T07:03:44.871 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T07:03:44.872 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T07:03:44.872 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=9747331 2024-02-05T07:03:44.872 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T07:03:44.872 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T07:03:44.872 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=0 2024-02-05T07:03:44.872 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T07:03:44.872 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T07:03:38.516618+0000' 2024-02-05T07:03:44.872 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T07:03:44.872 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T07:03:44.872 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=23 2024-02-05T07:03:44.872 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=27 2024-02-05T07:03:44.872 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T07:03:44.902 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 3 ~ false / 28 / 27 / true / 2024-02-05T07:03:38.516618+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T07:03:44.903 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T07:03:45.698 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T07:03:45.888 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T07:03:45.888 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=23 2024-02-05T07:03:45.888 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=28 2024-02-05T07:03:45.888 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T07:03:45.888 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T07:03:45.888 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T07:03:38.516' 2024-02-05T07:03:45.888 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=0 2024-02-05T07:03:45.888 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T07:03:38.516618+0000' 2024-02-05T07:03:45.888 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='0x0' 2024-02-05T07:03:45.888 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T07:03:45.888 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T07:03:45.888 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=9747331 2024-02-05T07:03:45.889 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T07:03:45.889 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T07:03:45.889 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=0 2024-02-05T07:03:45.889 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T07:03:45.889 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T07:03:38.516618+0000' 2024-02-05T07:03:45.889 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T07:03:45.889 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T07:03:45.889 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=23 2024-02-05T07:03:45.889 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=27 2024-02-05T07:03:45.889 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T07:03:45.918 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 4 ~ false / 28 / 27 / true / 2024-02-05T07:03:38.516618+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T07:03:45.918 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T07:03:46.715 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T07:03:46.912 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T07:03:46.912 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=23 2024-02-05T07:03:46.912 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=28 2024-02-05T07:03:46.912 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T07:03:46.912 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T07:03:46.913 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T07:03:38.516' 2024-02-05T07:03:46.913 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=0 2024-02-05T07:03:46.913 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T07:03:38.516618+0000' 2024-02-05T07:03:46.913 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='0x0' 2024-02-05T07:03:46.913 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T07:03:46.913 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T07:03:46.913 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=9747331 2024-02-05T07:03:46.913 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T07:03:46.913 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T07:03:46.913 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=0 2024-02-05T07:03:46.913 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T07:03:46.913 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T07:03:38.516618+0000' 2024-02-05T07:03:46.913 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T07:03:46.914 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T07:03:46.914 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=23 2024-02-05T07:03:46.914 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=27 2024-02-05T07:03:46.914 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T07:03:46.944 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 5 ~ false / 28 / 27 / true / 2024-02-05T07:03:38.516618+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T07:03:46.944 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T07:03:47.744 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T07:03:47.927 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T07:03:47.927 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=23 2024-02-05T07:03:47.927 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=28 2024-02-05T07:03:47.927 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T07:03:47.927 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T07:03:47.927 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T07:03:38.516' 2024-02-05T07:03:47.927 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=0 2024-02-05T07:03:47.928 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T07:03:38.516618+0000' 2024-02-05T07:03:47.928 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='0x0' 2024-02-05T07:03:47.928 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T07:03:47.928 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T07:03:47.928 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=9747331 2024-02-05T07:03:47.928 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T07:03:47.928 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T07:03:47.928 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=0 2024-02-05T07:03:47.928 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T07:03:47.928 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T07:03:38.516618+0000' 2024-02-05T07:03:47.928 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T07:03:47.928 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T07:03:47.928 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=23 2024-02-05T07:03:47.928 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=27 2024-02-05T07:03:47.929 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T07:03:47.957 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 6 ~ false / 28 / 27 / true / 2024-02-05T07:03:38.516618+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T07:03:47.957 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T07:03:48.746 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T07:03:48.933 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T07:03:48.934 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=23 2024-02-05T07:03:48.934 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=28 2024-02-05T07:03:48.934 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T07:03:48.934 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T07:03:48.934 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T07:03:38.516' 2024-02-05T07:03:48.934 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=0 2024-02-05T07:03:48.934 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T07:03:38.516618+0000' 2024-02-05T07:03:48.934 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='0x0' 2024-02-05T07:03:48.934 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T07:03:48.934 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T07:03:48.934 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=9747331 2024-02-05T07:03:48.934 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T07:03:48.934 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T07:03:48.935 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=0 2024-02-05T07:03:48.935 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T07:03:48.935 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T07:03:38.516618+0000' 2024-02-05T07:03:48.935 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T07:03:48.935 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T07:03:48.935 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=23 2024-02-05T07:03:48.935 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=27 2024-02-05T07:03:48.935 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T07:03:48.962 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 7 ~ false / 28 / 27 / true / 2024-02-05T07:03:38.516618+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T07:03:48.962 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T07:03:49.757 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T07:03:49.968 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T07:03:49.968 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=23 2024-02-05T07:03:49.969 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=28 2024-02-05T07:03:49.969 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T07:03:49.969 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T07:03:49.969 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T07:03:38.516' 2024-02-05T07:03:49.969 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=0 2024-02-05T07:03:49.969 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T07:03:38.516618+0000' 2024-02-05T07:03:49.969 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='0x0' 2024-02-05T07:03:49.969 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T07:03:49.969 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T07:03:49.969 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=9747331 2024-02-05T07:03:49.969 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T07:03:49.969 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T07:03:49.969 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=0 2024-02-05T07:03:49.969 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T07:03:49.970 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T07:03:38.516618+0000' 2024-02-05T07:03:49.970 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T07:03:49.970 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T07:03:49.970 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=23 2024-02-05T07:03:49.970 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=27 2024-02-05T07:03:49.970 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T07:03:49.998 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 8 ~ false / 28 / 27 / true / 2024-02-05T07:03:38.516618+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T07:03:49.998 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T07:03:50.795 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T07:03:51.057 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T07:03:51.058 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=23 2024-02-05T07:03:51.058 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=28 2024-02-05T07:03:51.058 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T07:03:51.058 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T07:03:51.058 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T07:03:38.516' 2024-02-05T07:03:51.058 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=0 2024-02-05T07:03:51.058 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T07:03:38.516618+0000' 2024-02-05T07:03:51.058 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='0x0' 2024-02-05T07:03:51.058 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T07:03:51.058 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T07:03:51.058 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=9747331 2024-02-05T07:03:51.058 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T07:03:51.058 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T07:03:51.059 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=0 2024-02-05T07:03:51.059 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T07:03:51.059 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T07:03:38.516618+0000' 2024-02-05T07:03:51.059 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T07:03:51.059 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T07:03:51.059 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=23 2024-02-05T07:03:51.059 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=27 2024-02-05T07:03:51.059 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T07:03:51.088 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 9 ~ false / 28 / 27 / true / 2024-02-05T07:03:38.516618+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T07:03:51.088 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T07:03:51.888 INFO:tasks.workunit.client.0.smithi148.stderr:dumped pgs 2024-02-05T07:03:52.082 INFO:tasks.workunit.client.0.smithi148.stdout:( 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[query_epoch]=23 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[query_seq]=28 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[query_active]=false 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule]='scrub scheduled' 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[query_schedule_at]='2024-02-06T07:03:38.516' 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_duration]=0 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_stamp]='2024-02-05T07:03:38.516618+0000' 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[query_last_scrub]='0x0' 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[query_is_future]=true 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[query_vs_date]=true 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[query_scrub_seq]=9747331 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_pg_state]='active+clean' 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_last_duration]=0 2024-02-05T07:03:52.083 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T07:03:52.084 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_schedule_at]='2024-02-06T07:03:38.516618+0000' 2024-02-05T07:03:52.084 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_is_future]=true 2024-02-05T07:03:52.084 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_vs_date]=true 2024-02-05T07:03:52.084 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_reported_epoch]=23 2024-02-05T07:03:52.084 INFO:tasks.workunit.client.0.smithi148.stdout:[dmp_seq]=27 2024-02-05T07:03:52.084 INFO:tasks.workunit.client.0.smithi148.stdout:) 2024-02-05T07:03:52.113 INFO:tasks.workunit.client.0.smithi148.stdout:--> loop: 10 ~ false / 28 / 27 / true / 2024-02-05T07:03:38.516618+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T07:03:52.113 INFO:tasks.workunit.client.0.smithi148.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T07:03:52.113 INFO:tasks.workunit.client.0.smithi148.stdout:WaitingAfterScrub : wait_any_cond(): failure. Note: query-active=false 2024-02-05T07:03:52.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:156: wait_any_cond: return 1 2024-02-05T07:03:52.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:521: TEST_just_deep_scrubs: return 1 2024-02-05T07:03:52.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: return 1 2024-02-05T07:03:52.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-02-05T07:03:52.114 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-test 1 2024-02-05T07:03:52.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T07:03:52.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-02-05T07:03:52.115 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T07:03:52.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T07:03:52.115 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T07:03:52.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T07:03:52.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T07:03:52.116 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T07:03:52.427 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T07:03:52.427 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T07:03:52.428 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T07:03:52.429 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T07:03:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T07:03:52.431 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T07:03:52.432 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T07:03:52.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T07:03:52.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T07:03:52.433 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T07:03:52.434 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T07:03:52.435 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T07:03:52.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-02-05T07:03:52.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-02-05T07:03:52.436 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-02-05T07:03:52.438 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-scrub-test/mgr.x.log td/osd-scrub-test/mon.a.log td/osd-scrub-test/osd.0.log td/osd-scrub-test/osd.1.log td/osd-scrub-test/osd.2.log /home/ubuntu/cephtest/archive/log 2024-02-05T07:03:52.439 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T07:03:52.453 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T07:03:52.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T07:03:52.454 INFO:tasks.workunit.client.0.smithi148.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.545011 2024-02-05T07:03:52.454 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.545011 2024-02-05T07:03:52.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T07:03:52.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T07:03:52.455 INFO:tasks.workunit.client.0.smithi148.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-02-05T07:03:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:03:52.456 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-02-05T07:03:52.457 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-05T07:03:52.749 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-test.sh) on smithi148 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh' 2024-02-05T07:03:52.975 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ce9c6d68acdb48f6a8f8ffc84e466fa1 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d4ed20cfb238ef2c47d64ca4533eca00b8968b98/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-test.sh) on smithi148 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh' 2024-02-05T07:03:52.978 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-05T07:03:53.006 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-05T07:03:53.007 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-05T07:03:53.051 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-05T07:03:53.059 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-05T07:03:53.060 DEBUG:teuthology.orchestra.run.smithi148:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T07:03:53.083 INFO:teuthology.orchestra.run.smithi148.stdout: remote refid st t when poll reach delay offset jitter 2024-02-05T07:03:53.083 INFO:teuthology.orchestra.run.smithi148.stdout:============================================================================== 2024-02-05T07:03:53.083 INFO:teuthology.orchestra.run.smithi148.stdout:*hv01.front.sepi 44.4.53.2 2 u 15 128 377 0.155 -2.499 0.415 2024-02-05T07:03:53.083 INFO:teuthology.orchestra.run.smithi148.stdout:+hv02.front.sepi 63.231.80.2 3 u 34 128 377 0.183 +0.380 0.291 2024-02-05T07:03:53.083 INFO:teuthology.orchestra.run.smithi148.stdout:+hv03.front.sepi 74.6.168.72 3 u 125 128 377 0.161 -2.845 0.326 2024-02-05T07:03:53.083 INFO:teuthology.orchestra.run.smithi148.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-02-05T07:03:53.084 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-05T07:03:53.093 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-05T07:03:53.093 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-05T07:03:53.102 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-05T07:03:53.112 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-05T07:03:53.120 INFO:teuthology.task.internal:Duration was 8408.513364 seconds 2024-02-05T07:03:53.120 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-05T07:03:53.129 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-05T07:03:53.130 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-05T07:03:53.170 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-05T07:03:53.171 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2024-02-05T07:03:53.171 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-05T07:03:53.227 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-05T07:03:53.228 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-05T07:03:53.312 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-05T07:03:53.312 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T07:03:53.410 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-05T07:03:53.420 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-05T07:03:53.421 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-05T07:03:53.433 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-05T07:03:53.462 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-05T07:03:53.487 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2024-02-05T07:03:53.510 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-05T07:03:53.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:03:53.563 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-05T07:03:53.583 INFO:teuthology.task.internal:Transferring archived files... 2024-02-05T07:03:53.585 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539668/remote/smithi148 2024-02-05T07:03:53.586 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-05T07:03:54.300 INFO:teuthology.task.internal:Removing archive directory... 2024-02-05T07:03:54.301 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-05T07:03:54.345 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-05T07:03:54.355 INFO:teuthology.task.internal:Not uploading archives. 2024-02-05T07:03:54.356 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-05T07:03:54.364 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-05T07:03:54.364 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-05T07:03:54.392 INFO:teuthology.orchestra.run.smithi148.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 5 07:03 /home/ubuntu/cephtest 2024-02-05T07:03:54.392 INFO:teuthology.orchestra.run.smithi148.stdout: 671146 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 5 04:52 /home/ubuntu/cephtest/mnt.0 2024-02-05T07:03:54.392 INFO:teuthology.orchestra.run.smithi148.stdout: 673172 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 5 06:48 /home/ubuntu/cephtest/mnt.0/client.0 2024-02-05T07:03:54.393 INFO:teuthology.orchestra.run.smithi148.stdout: 800836 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 5 07:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-05T07:03:54.393 INFO:teuthology.orchestra.run.smithi148.stdout: 800838 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Feb 5 07:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-02-05T07:03:54.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:03:54.394 INFO:teuthology.orchestra.run.smithi148.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-05T07:03:54.394 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-05T07:03:54.399 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-05T07:03:54.409 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-05T07:03:54.419 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi148 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-05T07:03:54.419 INFO:teuthology.nuke:Checking targets against current locks 2024-02-05T07:03:54.447 DEBUG:teuthology.nuke:shortname: smithi148 2024-02-05T07:03:54.447 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T07:03:54.470 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-01-31_07:15:30-rados-wip-nitzan-max-pglog-size-distro-default-smithi/7539668', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 04:34:49.541630', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEi9x4SB2ZkDPoRoPJVvBcM5XZ8KSrFdFl71VrJ7EllReZJGauGBsUzhTAR8jD1YjCMCg9xZfKb0PcnA6f3+8Tk='} 2024-02-05T07:03:54.519 INFO:teuthology.orchestra.console.smithi148:Power off 2024-02-05T07:03:54.520 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-05T07:03:54.543 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2024-02-05T07:03:54.543 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T07:03:54.566 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-02-05T07:03:58.568 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T07:03:58.678 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2024-02-05T07:04:02.680 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T07:04:02.790 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2024-02-05T07:04:02.891 INFO:teuthology.orchestra.console.smithi148:Power off completed 2024-02-05T07:04:02.996 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} duration: 8408.513363838196 failure_reason: 'Command failed (workunit test scrub/osd-scrub-test.sh) on smithi148 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d4ed20cfb238ef2c47d64ca4533eca00b8968b98 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh''' flavor: default owner: scheduled_nmordech@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ce9c6d68acdb48f6a8f8ffc84e466fa1 status: fail success: false 2024-02-05T07:04:02.996 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-05T07:04:03.071 INFO:teuthology.run:FAIL